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ABSTRACT 

This  study  examines  the  potential  impacts  of  the  DoD  emphasizing  the  use  of 
parametric  cost  estimating  methods  or  techniques  in  the  acquisition  process.  As  one 
of  many  initiatives  to  improve  the  DoD  acquisition  process  through  use  of 
commercial  practices,  parametric  cost  estimating  has  the  potential  to  be  helpful  in 
many  applications  for  which  it  has  never  before  been  considered.    This  study, 
conducted  through  a  questionnaire,  personal  interviews  and  a  review  of  recently 
released  publications  from  the  DoD  Joint  Government/Industry  Initiative,  identifies 
areas  of  interest  for  those  anticipating  using  parametric  cost  estimating  methods  and 
techniques.    These  areas  include  the  Program  Definition  &  Risk  Reduction  and 
Engineering  &  Manufacturing  Development  phases  of  program  management  as  well 
as  for  many  pre-award  contract  actions.    The  data  from  this  thesis  show  that  the 
majority  of  the  personnel  in  the  DoD  acquisition  community  believe  that  parametric 
cost  estimating  methods  can  be  used  effectively  in  those  areas.  The  data  also  show 
that  the  methods  also  may  have  applications  in  the  Production,  Fielding/Deployment 
&  Operational  Support  phase  of  program  management  and  post-award  contract 
actions  such  as  negotiating  changes,  forward  pricing  rate  agreements,  or  analyzing 
claims. 
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I .   INTRODUCTION 

A.  BACKGROUND 

This  thesis  analyzes  Department  of  Defense  (DoD)  Business 
Process  Reengineering  initiatives  that  focus  on  increasing  the 
use  of  parametric  cost  estimating  techniques  in  DoD 
applications.  Using  parametric  pricing  techniques  to  estimate 
program  costs  has  a  relatively  short  history  in  comparison 
with  other  pricing  techniques.  To  date,  cost  estimates  based 
on  parametric  estimating  methods  have  been  restricted  mainly 
to  overall  initial  planning  but  they  have  wider  applications 
that  are  now  being  contemplated  through  direct  policy  action. 
It  has  been  proposed  that  they  be  applied  to  price  estimates 
for  individual  contract  actions  or  families  of  contract 
actions  to  reduce  a  potential  contractor's  bid  and  proposal 
costs,  proposal  cycle  time,  and  to  expedite  the  Federal 
Government  proposal  evaluation  and  contract  award  process. 

B.  SCOPE  AND  ASSUMPTIONS 

The  objective  of  this  thesis  is  to  study  the  policy 
effects  of  increasing  the  use  of  parametric  cost  estimation 
methods  in  individual  contract  actions.  Any  reference  to 
program  management  uses  of  parametric  cost  estimating  methods 
is  done  to  relate  how  individual  contract  actions  fit  within 
the  framework  of  a  generic  program.  Anticipated  uses  in 
specific  programs  or  contract  actions  are  not  reviewed  or 


analyzed.  No  attempt  is  made  to  validate  any  existing 
parametric  cost  estimating  models,  techniques,  processes, 
software,  or  their  applications  in  specific  situations. 

This  thesis  identifies  potential  impediments  to  the 
implementation  of  the  policy  and  makes  recommendations  on  how 
they  can  be  overcome.  Specifically,  the  affects  on  proposal 
generation,  evaluation,  and  subsequent  contract  performance  is 
evaluated.  The  basic  effort  is  directed  at  producing  an 
accurate  picture  of  how  parametric  cost  estimating  techniques 
may  be  used  by  the  Department  of  Defense  (DoD)  and  contractors 
throughout  the  contracting  process  until  completion.  Many 
studies  have  been  done  on  the  use  of  parametric  cost  estimate 
development  for  entire  classes  of  equipments,  individual 
contract  actions,  or  programs,  but  none  have  focused  on  the 
affects  of  the  policy  behind  the  use  of  the  methods. 

It  was  assumed  at  the  onset  of  the  research  that  few 
organizations  and  personnel  employed  by  DoD  either  directly  or 
through  a  contract  vehicle  were  trained  in  the  use  of 
parametric  cost  estimating  methods.  It  was  further  assumed 
that  of  those  organizations  and  people  trained  in  their  use, 
most  would  be  concentrated  at  the  program  office  level  or 
higher  in  buying  organizations  or  in  the  DoD  policy  making 
areas.  Finally,  it  was  assumed  that  neither  the  Federal 
Government  as  a  whole  nor  DoD  as  a  subset  had  promulgated 


specific  guidance,  policy,  or  instructions  regarding  the  use 
and  application  of  parametric  cost  estimating  technigues. 

C.  RESEARCH  OBJECTIVE 

This  effort  is  intended  to  review  the  increased  attention 
being  paid  to  the  area  of  parametric  methods  in  .the  area  of 
cost  estimation  in  contracting.  This  effort  will  give  greater 
insight  into  the  potential  uses  for  parametric  cost  estimating 
techniques.  An  analysis  of  the  problems  that  may  be 
encountered  at  the  onset  of  policy  implementation  may  lead  to 
a  concerted  effort  to  meet  them  with  corrective  action  before 
they  become  impediments  in  program  management.  The  analysis 
provides  information  regarding  how  companies  contracting  with 
the  Federal  Government  and  Federal  Government  personnel 
interacting  with  these  companies  can  effectively  use 
parametric  cost  estimating  techniques  to  support  cost 
proposals  submitted  to  the  DoD. 

D.  RESEARCH  QUESTIONS 

Given  the  stated  objectives  for  this  thesis,  the  areas  of 
research  are: 

Primary  research  question: 

How  could  a  DoD  emphasis  on  using  parametric  methods  in 
cost  estimating  affect  the  procurement  process? 


Subsidiary  research  questions  include: 

1.  What  changes  could  offerors  make  to  their  proposal 
preparation,  submission,  and  support  processes  to 
accommodate  using  parametric  methods,  techniques,  or 
software  in  estimating  potential  contract  costs? 

2.  What  impacts  could  emphasis  on  using  cost  estimates 
based  on  parametric  estimating  techniques  have  on  source 
selection  criteria  and  on  the  evaluation  of  proposals  by 
the  DOD? 

3.  How  could  source  selection  criteria  and  evaluation 
factors  be  constructed  to  fairly  evaluate  a  proposal  that 
uses  a  parametric  cost  estimate  with  one  that  does  not? 

4.  What  effects  could  using  parametric  cost  estimations 
have  on  negotiations? 

E.    RESEARCH  METHODOLOGY 

The  basis  of  the  study  is  a  literature  review  through  the 
Defense  Logistics  Studies  Information  Exchange  (DLSIE) ,  the 
Defense  Technical  Information  Center  (DTIC) ,  and  various 
publications  and  journals.  Data  were  also  collected  through 
survey  questionnaires  from  Federal  Government  policy  offices, 
acquisition  management  organizations,  contracting  offices, 
companies  doing  business  with  the  Federal  Government,  and 
several  academic  institutions.  The  survey  used  is  provided  as 
Appendix  C  and  the  data  from  the  survey  are  presented  in 


Appendix  E.  Personnel  interviews  were  conducted  with  several 
representatives  from  DoD  organizations,  industry 
organizations,  and  companies  doing  business  with  the  DoD. 
Interview  questions  are  presented  in  Appendix  D  and  the 
information  from  the  interviews  is  incorporated  in.  the  body  of 
the  text. 
F.    ORGANIZATION  OF  THE  STUDY 

Chapter  I  of  this  thesis  is  an  introduction  providing  a 
brief  description  of  the  topic,  the  scope  of  the  research 
effort,  and  the  research  methodology  used.  A  review  of  the 
background  of  parametric  cost  estimating  is  discussed  in 
Chapter  II.  Cost  estimation  techniques  and  their  historical 
uses  in  Department  of  Defense  contracting  are  also  discussed. 
Chapter  III  describes  the  process  of  developing  a  parametric 
cost  estimate.  In  Chapter  IV,  the  data  collection  methodology 
is  described,  the  data  displayed,  and  data  analysis  presented. 
Chapter  V  concludes  the  research  effort  with  a  summary  of  the 
findings,  conclusions,  and  recommendations.  Appendices  will 
present  definitions  and  acronyms  used  and  the  survey  vehicle 
used  to  collect  the  data. 


II.   BACKGROUND  REVIEW  AND  HISTORICAL  PERSPECTIVE 

A.  PURPOSE 

The  purpose  of  this  chapter  is  to  lay  the  historical 
foundation  from  which  the  study  was  conducted.  Beginning  with 
the  development  of  parametric  cost  estimating  techniques,  the 
chapter  next  discusses  their  application  in  the  source 
selection  segment  of  the  Federal  Government's  acquisition 
process.  Parametric  cost  estimating  techniques  are  defined, 
as  well  as  several  other  terms  relating  to  the  acquisition 
process,  as  they  relate  to  the  use  of  parametric  cost 
estimating  methods  or  techniques.  Potential  applications  in 
both  Federal  Government  acquisition  and  commercial  arenas  are 
examined.  Finally,  the  chapter  closes  by  discussing  the 
advantages  and  limitations  of  using  parametric  cost  estimating 
techniques.  This  information  provides  the  background  for 
understanding  why  there  is  an  interest  in  applying  the 
techniques  more  widely  than  they  have  been  in  the  past. 

B .  BACKGROUND 
1.    History 

Historically,  the  use  of  parametric  cost  estimates  for 
pricing  purposes  is  relatively  new.  Whereas  price  analysis 
is  practiced  in  some  form  for  every  purchase  that  has  been 
made,  the  techniques  used  in  parametric  cost  estimating  have 


their  origins  in  the  learning  curve  theory  espoused  by  T.P. 
Wright  for  use  in  the  aircraft  industry  in  the  latter  part  of 
the  1930s.  [Ref  37:p.4]  The  common  sense  notion  that 
production  of  an  item  becomes  more  efficient  as  the  quantity 
produced  increased  was  finally  quantified.  Having  this 
relationship  mathematically  derived  rather  than  estimated 
through  experience  and  guesswork,  allowed  manufacturers  to 
begin  forecasting  requirements  more  accurately  over  several 
production  runs.  Managers  could  predict  the  cost  of  resources 
needed  to  produce  or  the  delivery  schedule  for  a  given  item  at 
a  given  time  in  the  production  cycle  through  the  use  of  a 
logarithmic  graph  of  units  of  production  and  labor  hours 
needed  to  produce  a  given  unit  of  production.  [Ref  37: p. 4] 

Following  World  War  II,  other  estimating  techniques  were 
developed  and  refined.  Wessel,  in  his  article,  describes 
several  methods  in  use  at  that  time  to  rapidly  estimate 
project  costs  for  use  in  comparative  analysis  of  capital 
investment  opportunities.  [Ref  46:p.l69]  Of  interest  to  the 
development  of  parametric  cost  estimating  is  the  discussion  of 
cost  per  ton  of  product  in  the  U.S.  chemical  industry. 
Although  this  is  certainly  not  the  first  time  ratio  analysis 
was  used  in  industry,  it  is  an  indicator  that  the  technique 
was  in  common  use.  Attempts  were  also  being  made  at  that  time 
to  increase  the  accuracy  of  the  estimates  through  analyzing 


the  data  and  correlating  costs  to  several  production  factors. 
At  that  time,  the  estimates  were  said  to  be  accurate  enough 
to  produce  results  which  allowed  managers  to  narrow  the  choice 
of  new  products  or  processes  to  the  most  competitive 
available. 

Descriptions  of  several  common  estimating  techniques  used 
by  commercial  industry  in  the  late  1950s  are: 


1.  Detailed  Estimates  are  individual  cost  elements 
compiled  from  in-depth  engineering  projections. 

2.  Preliminary-Bill  Estimates  are  materials  priced  from 
price  lists  or  quotations  and  labor  costs  derived  from 
labor-material  ratios  seen  on  similar  jobs  in  the  past. 

3.  Layout  Estimates  are  equipment  costs  obtained  from 
vendor  quotations  or  price  lists  and  then  corrected  for 
various  inflation  factors  using  appropriate  indices. 
Material  costs  are  estimated  based  on  some  cost  per  unit 
of  output  basis,  with  engineering  costs  calculated  from 
a  percentage  curve  based  on  past  experience  with  similar 
projects . 

4.  Equipment-Ratio  Estimates  are  estimated  by  using 
ratios  developed  from  experience  on  past  projects. 

5.  Capacity-Cost  Estimates  are  estimates  made  by 
projecting  costs  for  a  new  plant  by  using  a  logarithmic 
plot  of  similar  plant  capacities  and  costs  from 
historical  records.  [Ref  14:p.l31] 


These  techniques  show  increasing  interest  in  producing 
accurate  cost  estimates  and  from  that  interest  greater 
sophistication  in  generating  those  estimates.  While  there 
were  elements  of  these  techniques  that  are  necessary  for 
parametric  cost  estimating,  they  had  not  yet  come  together. 
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More  importantly,  they  did  lay  the  foundation  for  parametric 
cost  estimating.  To  produce  estimates  using  many  of  these 
methods  the  way  data  were  collected,  analyzed,  displayed,  and 
stored  had  to  be  changed. 

Concurrent  with  the  interest  in  the  U.S.  commercial 
industry  with  greater  estimate  accuracy,  the  Federal 
Government,  particularly  the  DoD,  was  under  pressure  to 
develop  more  accurate  estimates.  As  in  industry,  the  Rand 
Corporation  noticed  relationships  between  costs  and  production 
outputs.  [Ref  37: p. 7]  Whereas  industrial  applications  related 
costs  to  production  quantities  or  consumption  of  raw  materials 
used,  the  Rand  Corporation  analyzed  the  correlations  between 
costs  and  variables  that  described  operating  characteristics 
of  aircraft.  In  particular,  it  was  observed  that  certain 
classes  of  aircraft  had  similar  cost  behaviors  regarding 
weight,  speed,  range,  and  altitude.  These  behaviors  were 
called  Cost  Estimating  Relationships  (CER)  and  are  the  basis 
for  parametric  cost  estimating.  The  combination  of  learning 
curve  theory,  available  data  from  extensive  production  of 
similar  items  using  similar  processes  (aircraft  production 
during  and  after  World  War  II),  and  development  of  the  idea  of 
the  CER  came  together  in  the  form  of  parametric  cost 
estimating. 
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The  parametric  estimation  methods  have  been  used  for  many 
years  with  ever  widening  applications  being  found  for  their 
use.  From  some  of  the  original  uses  in  the  Department  of  the 
Navy  (DoN)  as  first  order  estimates  relating  aircraft  and  ship 
weight  and  cost  to  develop  simple  and  fairly  reliable  cost 
estimates,  the  methods  have  been  widely  applied.  Parametric 
cost  estimating  allowed  planners  to  make  accurate  estimates  of 
system  costs  at  early  stages  in  design,  a  solution  recognized 
in  both  commercial  and  Federal  Government  applications.  The 
Secretary  of  Defense  (SECDEF)  became  convinced  of  the  utility 
of  Independent  Parametric  Cost  Estimates  (IPCE)  in  1971  as  a 
tool  to  control  cost  growth  in  major  weapon  systems.  [Ref 
25:p.l2]  He  issued  directives  to  use  the  techniques  in  cost 
estimating  at  the  program  level  to  solve  the  problem  of 
accurately  estimating  Research,  Development,  Test  and 
Evaluation  (RDT&E)  and  initial  production  run  costs. 
Estimation  difficulties  in  the  early  stages  of  production  at 
that  time  were  exacerbated  by  the  rapidly  escalating  pace  of 
technological  development.  It  was  felt  that  success  in  the 
early  stages  of  a  program's  life  would  lead  to  reductions  in 
program  cost  overruns.  In  conjunction  with  management 
difficulties  caused  by  cost  overruns,  there  was  enormous 
pressure  to  reduce  the  DoD  budget.  To  manage  these  budgetary 
pressures,  SECDEF  hoped  to  have  each  Service  perform  IPCEs  at 
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each  key  decision  point  in  the  weapons  acquisition  process  to 
increase  the  accuracy  of  cost  estimates. 

Since  the  inception  of  the  idea  of  parametric  cost 
estimating  and  study  by  the  Rand  Corporation,  there  has  been 
increasing  application  in  commercial  industry  and.  the  Federal 
Government  for  planning  purposes.  Initial  uses  of  parametric 
cost  estimates  centered  mainly  on  [Ref  20:p.l2] 


1.  Identifying  possible  cost/performance  tradeoffs  in  the 
design  effort. 

2.  Providing  a  basis  for  cost/effectiveness  reviews  of 
performance  specifications. 

3.  Providing  information  useful   in  the  ranking  of 
competing  alternatives. 

4.  Suggesting  a  need  for  identifying  and  considering  new 
alternatives . 


The  reasonably  accurate  estimates  in  the  early  stages  of 
program  development  and  widespread  applicability  contributed 
greatly  to  the  acceptance  of  parametric  cost  estimating  as  a 
valid  method.  Increasing  competitive  forces  have  encouraged 
their  use  in  commercial  applications  while  high  level  interest 
and  greater  public  scrutiny  have  led  to  wider  use  in  DoD 
areas . 

Figure  1  describes  a  general  relationship  between 
procurement  cost  estimating  techniques  used  and  the  time  line 
throughout  a  DoD-managed  program.   For  comparison  purposes, 
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the  cost  estimating  techniques  used  before  the  advent  of 
parametric  cost  estimating  are  shown  along  the  base  of  the 
figure  where  they  might  have  been  used  at  various  times  during 
the  program  life.  Parametric  cost  estimates  are  used  in  all 
phases  of  a  program  life  cycle  but  are  most  often  the  basis  of 
estimates  in  the  Program  Definition  and  Risk  Reduction 
(PD&RR) (Phase  I)  of  a  program  life  cycle.  [Ref  26:p.21] 
Parametric  cost  estimating  is  perceived  as  the  best  method  of 
generating  a  detailed  estimate  from  the  available  data  when 
only  mission  and  performance  parameters  have  been  described. 
[Ref  23:p.291] 


Phase  O 
CE 


EO 


Cos 


p 

For 
Budgeting 


Phase  I 
PD&RR 


t  Uncertainty  Band 


P    compared 
To  A 


Phase  II 

E&MD 


BU  BU  BU 

Initial       Compared         LRIP 
Design        To  P  Validated 


Phase  III 

PF/DOS 


BU  based  on  production 
Experience  (Compared  to  P) 


Capacity  Cost 


Equipment  Ratio  Layout  Estimates 


Preliminary  Bill  Detailed 


EO  =  Expert  Opinion  P  =  Parametric     A  =  Analogy      BU  =  Bottoms    Up    LRIP  =  Low  Rate    Initial 

Production 

Figure  1:  Program  Phases  and  Types  of  Estimates 


Source:  Refs  22,  25,  and  33 
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From  its  inception,  parametric  cost  estimating  has 
experienced  an  ever  expanding  base  of  uses.  Aircraft  programs 
have  long  been  the  beneficiaries  of  parametric  price 
estimating  techniques.  Weight  has  been  the  primary 
independent  variable  to  parametrically  estimate  costs  for 
aircraft  systems.  Analysis  by  the  Rand  Corporation  showed 
that  another  useful  variable  that  shows  a  good  correlation  to 
cost  is  speed.  [Ref  20:p.21]  Other  analyses  have  defined  a 
parametric  relationship  between  cost  and  wetted  area,  aspect 
ratio,  number  of  engineering  drawings,  number  of  parts,  and 
even  time  in  aircraft  development  and  production  programs. 
[Ref  20:p.34]  Despite  all  of  the  analysis,  weight  is  the  most 
consistent  and  accurate  parameter  to  estimate  aircraft  program 
costs.  [Ref  20:p.36]  No  longer  applied  solely  to  Federal 
Government  aviation  programs,  the  techniques  have  been  used  in 
a  wide  variety  of  Federal  Government  areas.  Projects  in  the 
Federal  Government  that  have  used  parametric  cost  estimating 
techniques  include  Department  of  Energy  construction  of  dams 
and  mining  structures.  Commercial  uses  range  from 
construction  and  aircraft  manufacturing  to  computer  software 
generation  projects. 

The  DoN  is  heavily  dependent  on  the  use  of  parametric 
pricing  for  major  systems  planning.  It  forms  a  major  part  of 
shipbuilding   and  aircraft   cost   estimating  both  at   the 
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aggregate  level  (program)  and  equipment  level.  The  techniques 
are  used  in  projections  in  shipbuilding  to  estimate  costs 
within  each  element  of  the  Expanded  Ship  Work  Breakdown 
Structure.  Similar  to  the  CERs  observed  from  aircraft 
production,  one  of  the  most  useful  correlations  is  that  of  a 
ship's  gross  tonnage  to  cost.  [Ref  35:p.23]  One  of  a  ship 
class's  most  consistent  physical  aspects  that  shows  a  useful 
correlation  to  cost  is  that  of  gross  tonnage,  making  it  a  most 
useful  variable  in  ship  cost  estimation.  Other  CERs  that  have 
been  developed  and  used  successfully  for  shipbuilding 
estimates  are  man-hours  to  gross  tonnage  and  power  rating 
ratios  to  gross  tonnage  relationships.  [Ref  35:p.37] 

The  development  and  use  of  parametric  cost  estimating  has 
followed  that  of  the  development  of  computing  capability. 
Prior  to  the  widespread  availability  and  use  of  computers, 
data  collection  and  analysis  on  the  scale  required  for 
parametric  cost  estimating  was  prohibitively  expensive  and 
time  consuming.  As  computer  hardware  improved  so  has  the  use 
of  parametric  cost  estimating.  Software  has  also  made 
tremendous  improvements  that  parallel  those  of  hardware.  More 
sophisticated  processing  of  data  and  user  friendly  programs 
have  combined  with  the  hardware  improvements  to  bring 
parametric  cost  estimating  into  common  use.  These 
capabilities  allow  quantitative  analysis  of  such  accumulations 
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of  data  that,  without  the  computer,  parametric  cost  analysis 
would  not  be  useful  in  terms  of  timeliness  even  if  it  were 
possible  at  all  in  a  manual  environment. 

2.    Source  Selection  Process  Development 

Parametric  cost  estimating  occupies  an  important  niche  in 
the  source  selection  process.  For  the  contractor  proposing  to 
do  business  with  the  Federal  Government,  estimate  accuracy 
could  mean  the  difference  between  winning  and  losing  the  award 
of  the  contract.  A  contractor  who  uses  a  poor  estimate  in  a 
Firm  Fixed-Price  (FFP)  contract  environment  will  suffer  by 
using  that  estimate.  He  will  either  lose  money  through 
underbidding  his  costs  if  he  bids  lower  than  his  actual  costs 
and  wins  the  contract  award,  or  he  will  lose  business  if  his 
estimate  is  too  high,  he  bids  too  high,  and  he  is  not  awarded 
the  contract.  For  the  Federal  Government,  a  reliance  on  a  low 
estimate  could  lead  to  quality  control  problems.  The 
contractor  who  won  the  contract  with  a  low  bid  will  attempt  to 
limit  his  loss  through  contractor  cost  cutting  measures.  In 
a  Cost-Reimbursement  contract  environment,  additional 
management  resources  will  be  required  of  the  Government  to 
control  cost  overrun  situations.  The  increased  costs  will 
also  bring  greater  public  scrutiny  on  both  the  Government  and 
the  contractor  as  cost  overruns  develop.  In  either  case, 
inaccurate  cost  estimates  damage  the  competitive  industrial 
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base  and  have  adverse  impacts  on  the  Federal  Government 
procurement  process. 

Along  with  the  greater  computing  capacity,  the  growth  of 
parametric  cost  estimating  depended  on  the  data  that  were 
available  for  analysis.  Other  Federal  Government  applications 
depended  on  the  collection  of  data  for  analysis.  This  data 
collection  was  made  available  in  large  part  through  dependence 
on  detailed  engineering  cost  and  schedule  estimates.  The 
Truth  in  Negotiations  Act  of  1962  (TINA)  codified  cost  and 
pricing  data  requirements  that  were  to  be  submitted  in 
contract  negotiations  with  the  Federal  Government.  In  meeting 
these  requirements  with  detailed  cost  estimates,  commercial 
industry  built  the  databases  required  for  effective  parametric 
cost  estimating. 

Cost  Accounting  Standards  (CAS)  has  also  had  a 
significant  impact  on  the  use  of  parametric  cost  estimating 
methods.  One  of  the  basic  premises  of  CAS  is  consistency  in 
estimating,  accumulating  and  reporting  costs  by  a  company 
proposing  to  do  business  with  the  Federal  Government.  Any 
parametric  model  used  by  a  company  must  be  able  to  segregate 
estimates  of  costs  to  match  the  data  collection  system  used  in 
that  company.  This  in  turn  should  match  the  proposed  work 
breakdown  structure  (WBS)  upon  which  understanding  of  the 
contractor's  ability  to  perform  work  delineated  in  the 
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Statement  of  Work  (SOW)  depends.  Failures  in  this  area  could 
make  it  difficult  for  a  buying  organization,  the  Defense 
Contract  Audit  Agency  (DCAA) ,  or  the  Defense  Contract 
Management  Command  (DCMC)  to  evaluate  a  company's  cost 
proposal.  Differences  here  could  also  result  in  CAS 
violations  which  in  turn  could  lead  to  adverse  effects  on 
prior  performance  ratings. 

Cost  estimating  plays  a  major  part  of  any  procurement 
decision  from  influencing  the  budgetary  processes  to  final 
source  selection.  The  Competition  in  Contracting  Act  (CICA) 
of  1984  followed  TINA,  and  both  had  tremendous  impacts  on  the 
source  selection  process  in  Federal  Government  procurement. 
Where  TINA  determined  data  to  be  submitted  as  the  basis  for  a 
cost  proposal,  CICA  clearly  allowed  for  negotiated  procedures 
in  a  competitive  environment  for  Federal  Government 
procurement  when  sealed  bidding  conditions  do  not  apply.  [Ref 
41]  Beyond  the  FAR  and  Defense  Federal  Acquisition  Regulation 
Supplement  (DFARS)  guidance,  executive  agencies  have  published 
additional  regulatory  constraints  that  affect  the  conduct  of 
Federal  Government  procurement  actions.  General  guidance  is 
given,  however,  in  the  DCAA  Contract  Audit  Manual.  [Ref 
37: Appendix  C]  Parametric  cost  estimating  techniques  are  not 
specified  for  use  or  directed  to  be  used  in  any  of  this 
guidance;   rather   their   use   has   grown   as   a   sensible, 
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businesslike  answer  to  meet  the  needs  of  individual  programs 
and  procurement  actions.  [Ref  37 :p. 7, 198] 

In  acquisition  planning,  specified  in  CICA  and  depicted 
in  Figure  1,  parametric  cost  estimating  is  applied  to  provide 
a  manager  with  useful  data  upon  which  he  can  make  an  informed 
decision.  Source  selection  planning  for  a  particular 
procurement  is  only  a  part  of  the  overall  plan,  but  it  is  the 
place  where  the  uses  of  parametric  cost  estimating  are  best 
illustrated.  An  individual  procurement  action  using 
parametric  cost  estimating  can  occur  in  any  of  the  four  phases 
shown  in  Figure  1.  Determining  that  parametric  cost 
estimating  is  useful  in  the  conduct  of  a  procurement  action 
may  have  the  most  beneficial  outcome  when  used  in  the  planning 
stages.  Parametric  cost  estimating  considerations  can  best 
be  integrated  into  procurement  planning  to  fully  exploit  their 
potential . 

Use  of  parametric  cost  estimating  techniques  could  first 
impact  the  individual  procurement  by  affecting  the  structure 
of  the  solicitation  for  an  item  or  service.  The  buying 
activity  may  choose  to  construct  the  solicitation  to  accept, 
reject  or  encourage  the  use  of  parametric  cost  estimates. 
Requiring  the  use  of  parametric  cost  estimating  techniques  may 
in  some  instances  restrict  competition  to  those  with  the 
experience  in  not  only  providing  the  goods  or  services 
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required  but  also  knowledge  and  experience  with  the  techniques 
specified.  Rejecting  the  use  of  parametric  cost  estimating 
techniques  as  valid  methods  of  supporting  a  price  proposal 
could  remove  a  competitive  advantage  for  a  company  in  another 
situation. 

From  the  solicitation,  the  companies  submitting  proposals 
will  construct  their  proposals  in  a  way  to  present  their 
company  as  the  best  value  for  the  buying  organization.  They 
will  present  the  data  required  by  the  solicitation  in  support 
of  their  proposal.  For  those  companies  familiar  with 
parametric  cost  estimating  techniques,  this  could  be  a  direct 
competitive  advantage.  Conversely,  for  those  who  are  not 
familiar  with  them  it  could  be  an  administrative  burden  at 
best  and  a  liability  in  the  competitive  environment  if  they 
are  required  by  the  buying  agency.  The  buying  organization 
should  recognize  the  differences  in  how  parametric  cost 
estimates  are  developed  from  the  traditional  bottoms  up 
estimates  and  structure  the  support  data  requirements 
accordingly.  In  either  situation,  the  buying  organization 
should  restrict  the  data  to  be  submitted  to  only  those 
required  to  support  analysis  of  the  proposal. 

The  evaluation  criteria  for  the  procurement,  expressed  in 
the  solicitation,  should  drive  the  data  collection  efforts. 
These  criteria  are  required  to  be  explained  in  the  body  of  the 
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solicitation  to  potential  offerors.  Companies  proposing  to  do 
business  with  the  Federal  Government  will  use  these  criteria 
as  considerations  in  the  construction  of  their  proposal.  Each 
buying  organization  may  have  a  different  view  of  the 
interrelationships  between  the  elements  in  the  evaluation 
criteria.  Parametric  cost  estimating  could  have  impacts  not 
only  in  the  area  of  cost  comparisons  and  cost  realism  but  also 
in  the  technical  proposal  involving  understanding  the 
requirement.  In  the  evaluation  of  management  capability, 
parametric  cost  estimating  could  again  demonstrate  a  company' s 
level  of  understanding  of  the  requirement  as  well  as  affect  a 
company's  score  on  past  performance. 

The  proliferation  of  data  collection  systems,  management 
systems,  and  user  friendly  statistical  analysis  programs  make 
the  techniques  of  parametric  cost  estimating  more  available 
than  ever  before.  The  techniques  have  generated  an  industry 
whose  goal  is  to  provide  parametric  estimating  services  and 
tools  to  potential  users.  The  DoD-maintained  Constructive 
Cost  Model  (COCOMO)  was  developed  to  estimate  software  cost 
estimates.  One  of  the  more  widely  used  parametric  cost 
estimating  models  produced  by  commercial  industry  is  the 
Programmed  Review  of  Information  for  Costing  and  Evaluation 
(PRICE) .  [Ref  37:p.219]  The  PRICE  model  was  initially 
developed   by   the   RCA  Corporation   in   1974   to   provide 
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engineering  and  manufacturing  costs  for  equipment  having  a 

wide   range   of   applications.   It   has   undergone   several 

enhancements  since  its  inception  and  is  now  a  family  of  cost 

estimating  models  for  hardware  and  software  projects.  [Ref 

21:p.l45]    PRICE,   however,   is  only  one  of  many  models 

developed  by  industry  and  the  Federal  Government  to  meet  the 

needs  of  program  and  procurement  management. 

The   development   of  parametric   cost   estimating  has 

progressed  along  with  increasing  budgetary  pressures,  and 

technical  and  management  improvements.   The  current  state  of 

the  art  in  all  of  these  areas  has  brought  renewed  interest  in 

their  use  by  the  upper  echelons  of  DoD.   As  a  step  beyond 

their  earlier  uses,  the  present  drive  is  focused  on  reduction 

of  contractor  costs  from  onerous  Federal  Government  reporting 

requirements  and  an  intense  desire  to  reduce  equipment 

development  cycle  times. 

One  way  to  reduce  the  burden  associated  with 
providing  cost  and  pricing  data  is  to  permit 
greater  use  of  parametric  estimating  techniques. 
Properly  calibrated  parametric  techniques  can 
accurately  estimate  costs  while  reducing  bid  and 
proposal  costs  and  proposal  cycle  time  and 
expediting  the  Government  evaluation  process . [Ref . 
42:p.l] 

To  reach  the  goal  set  out  in  the  above  memorandum  by  the 

Under  Secretary  of  Defense,  the  use  of  parametric  pricing  will 

have  to  become  more  widely  accepted  throughout  the  DoD 
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acquisition  community  as  a  cost  estimation  tool  rather  than 
viewed  as  a  program  management  tool  for  cost  estimates. 
C.    DEFINITIONS  AND  CONCEPTS 

1.  Parametric  Cost  Estimating 

Parametric  cost  estimating  involves  the  collection  and 
organization  of  historical  information,  analysis  through 
mathematical  techniques,  and  relating  this  information  to  the 
work  output  that  is  being  estimated.  A  parametric  cost 
estimate  can  be  defined  as  an  estimate  which  predicts  cost  by 
means  of  explanatory  variables  such  as  performance 
characteristics,  physical  characteristics,  and  characteristics 
relevant  to  the  development  process,  as  derived  from 
experience  on  logically  related  systems.  [Ref  37:p.l72, 
22:p.293]  Combinations  of  CERs  and  their  relationships  are 
used  to  describe  the  likely  cost  behavior  of  a  project.  The 
emphasis  in  parametric  cost  estimating  is  to  "focus  on  the 
cost  drivers,  not  the  miscellaneous  details"  [Ref  37:"p.ll] 
that  are  common  in  other  cost  estimating  techniques. 

2 .  Parametric  Cost  Model 

A  parametric  cost  model  is  simply  the  group  of  CERs  and 
logical  relationships  used  together  to  produce  cost  estimates. 
They  can  range  from  common  rules  of  thumb,  such  as  costs  per 
gallon,  to  elaborate  computer  models.  [Ref  17:p.l2]  A 
parametric  cost  model  is  one  derived  from  the  relationships 
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between  data  points  in  a  historical  database  used  to  provide 
a  top  down  cost  estimate.  The  buyer  is  attempting  to  relate 
what  he  or  she  wants  to  buy  in  units  of  capability  to  the  cost 
for  it.  This  is  a  departure  from  the  traditional  industrial 
engineering  (or  bottoms  up)  approach  of  describing  what  a 
buyer  is  paying  for  through  the  use  of  detailed  cost 
accumulation  systems. 

Parametric  price  analysis  is  used  at  all  stages  of  a 
major  procurement  from  the  initial  estimates  to  developing 
estimates  for  costs  for  components  later  in  production. 
Figure  1  is  a  representation  of  how  parametric  pricing  is 
typically  used  in  comparison  with  other  cost  estimating 
methods. 

3.    Cost  Estimating  Relationships 

The  CER  is  the  key  component  in  the  development  of  a 
parametric  cost  estimating  model.   CERs  are  described  as: 


...an  algorithm  relating  the  cost  of  an  element  to 
physical  or  functional  characteristics  of  that  cost 
element  or  a  separate  cost  element;  or  relating  the 
cost  of  one  cost  element  to  the  cost  of  another 
cost  element.  [Ref  37:p.l65] 


This  idea  is  rooted  in  the  fact  that  costs  in  the  production 
of  an  item  are  related  in  a  quantifiable  manner  to  its 
physical  or  performance  characteristics.  An  example  of  this 
principle  in  the  Contract  Pricing  Resource  Guides   (CPRG) 
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describes  the  procedure  of  developing  a  CER  for  building 
houses  and  relating  the  costs  to  such  variables  as  the  number 
of  bathrooms,  available  living  area,  exterior  wall  surface,  or 
roof  surface  area.  [Ref  l:p.D-31] 

CERs  can  be  broken  down  into  two  categories,  cost- to-cost 
(overhead  rates)  and  cost  to  non-cost  (cost  per  pound  of 
thrust)  .  It  is  critical  that  the  cost  analyst  understand  the 
logical  relationship  between  cost  and  any  independent  variable 
for  the  model  to  be  used  effectively.  This  understanding 
leads  to  knowing  the  limits  of  the  model  and  potential  effects 
of  using  it  with  a  new  database. 

4 .  Mathematical  Techniques  Used  in  CER  Development 
Regression  analysis  and  multiple  regression  analysis  are 
two  statistical  techniques  that  can  be  used  to  relate  the  cost 
(dependent  variable)  to  a  characteristic  (independent 
variable).  [Ref  23:p.291]  The  aim  of  regression  analysis,  in 
the  form  of  applying  linear,  curvilinear,  logarithmic,  or 
exponential  techniques  to  the  data,  is  to  derive  the  equation 
of  a  curve  that  predicts  how  an  independent  variable  will  vary 
with  respect  to  a  given  parameter  or  dependent  variable.  In 
the  case  of  procurement,  interest  has  been  focused  on  the  cost 
and  schedule  estimating  aspects.  When  the  parametric 
relationship  has  been  described,  it  can  be  used  to  estimate 
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system  costs  by  substituting  different  cost,  performance,  or 
other  characteristics  into  the  equation. 

A  second  mathematical  method  that  can  be  used  to  derive 
a  CER  is  the  Least  Squares  Best  Fit  (LSBF)  method.  [Ref 
37:p.41]  This  algebraic  method  minimizes  the  sum  of  the 
squared  deviations  of  the  observed  values  of  a  variable  and 
calculated  values  of  that  variable.  In  this  method,  a  series 
of  calculations  involving  the  mean  values  of  the  dependent  and 
independent  values  and  the  point-slope  formula  are  used  to 
arrive  at  a  formula  that  describes  the  CER  between  the  two 
variables . 

Apart  from  strict  mathematical  derivation  of  the 
relationships  between  two  variables,  plotting  a  series  of  data 
points  and  graphically  determining  the  line  of  "best  fit"  can 
also  be  used  to  derive  a  CER.  [Ref  37:p.41]  This  method 
requires  plotting  the  data  points  and  simply  drawing  a  line 
through  the  data  points  leaving  an  equal  number  of  points  on 
either  side  of  the  line.  The  drawn  line  must  follow  the 
general  trend  of  the  data  points  and  every  data  point  must  be 
considered.  Any  outlying  data  points  must  not  be  discarded 
but  instead  investigated  to  understand  why  the  variation  in 
the  data  occurred.  The  point-slope  algebraic  formula  for  a 
line  can  be  used  to  derive  the  CER  in  this  method  if  a  linear 
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relationship  is  noted.   Curvilinear  regression  is  another 
method   available  to  develop  the  CER. 

5.    Statistical  Measures  of  Accuracy 

The  statistical  measures  of  the  accuracy  of  an  estimate 
based  on  the  two  described  mathematical  methods  can  be 
described  in  terms  of: 

1.  The  Standard  Error  of  the  Estimate  (SSE)  -  the 
variance  associated  with  the  prediction  made  from 
the  estimating  equation.  [Ref  25:p.l3] 

2.  The  Coefficient  of  Determination  -  the  closeness 
of  fit  between  two  variables  or  how  much  one  factor 
is  affected  by  another.  [Ref  37:p.46] 

With  these  measures,  a  manager  has  another  tool  to  assess 
the  applicability  of  the  CER  to  a  given  program  or  system. 
Using  regression  analysis,  the  data  are  then  analyzed  for 
relationships.  The  smaller  the  SSE  and  the  larger  the 
Coefficient  of  Determination  the  more  accurate  the  equation 
will  estimate  the  costs  that  will  be  incurred.  Examination  of 
the  estimates  made  by  the  equations  over  the  data  range  is  a 
critical  step  in  the  evaluation.  Often  there  are  breaks  or 
discontinuities  observed  in  the  data.  At  these  points  the 
relationship  between  one  independent  variable  and  the 
dependent  variable  breaks  down  and  another  relationship 
between  the  dependent  variable  and  a  second  independent 
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variable  becomes  a  more  reliable  predictor.  There  may  be 
several  CERs  for  a  given  item  or  system  depending  on  the  level 
of  production  or  range  of  a  given  performance  characteristic. 

6.    Model  Validity 

Comparability  between  the  items  or  services  upon  which 
the  database  is  founded  and  those  to  be  estimated  is  an  idea 
even  more  basic  to  parametric  cost  estimating  than  the  CER. 
Hertling  defines  parametric  cost  estimating  model  validation 
as  "the  process  of  determining  that  a  model  is  a  reliable 
predictor  of  costs  for  the  type  of  system  being  estimated." 
[Ref  17:p.l3]  This  is  also  an  issue  in  the  analog  method  of 
cost  estimating.  Analog  cost  estimating  methods  are  simply 
comparisons  between  past  costs  observed  on  similar  types  of 
systems  or  equipments  and  new  ones.  Although  there  may  be 
some  difficulty  in  determining  what  are  logically  related 
systems  or  programs  that  can  be  compared,  the  techniques  can 
be  easily  applied  if  such  a  relationship  can  be  defined 
between  an  existing  system  or  program  and  a  projected  one.  As 
in  the  analog  methods  of  cost  estimating,  the  greater  the 
difference  between  the  programs,  systems,  or  equipments  that 
make  up  the  database  and  the  projected  system,  the  less  useful 
is  the  estimate  produced. 

The  validity  of  the  historical  data  analyzed  is  one  of 
the  most  important  factors  in  its  later  applications.  [Ref 
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37:p.l2]  Data  input  to  the  model  must  be  verifiable  and 
accessible  to  the  cost  estimator.  Data  accuracy  is  critical 
and  has  more  dimensions  than  just  similarity  between  the  old 
and  new  programs,  systems,  or  equipments.  Parametric  cost 
estimates  are  least  useful  where  there  are  few  data  points  for 
development  of  the  basis  for  the  CERs  or  when  the  methods  of 
performing  a  task  change  significantly.  The  comparability 
between  a  new  system  and  existing  ones  whose  costs  make  up  the 
database  used  affect  the  model's  accuracy.  Accuracy  increases 
when  the  methods  used  for  production  of  like  items  or  systems 
are  very  similar.  An  example  of  this  is  a  system  that  has 
undergone  upgrades,  with  the  new  item  or  system  incorporating 
technological  innovations  or  increased  performance 
characteristics  beyond  those  previously  attained.  [Ref 
23:p.293] 

7.    Model  Validation  and  Calibration 

One  method  of  validating  a  parametric  cost  estimating 
model  involves  generating  estimates  with  the  model  for 
comparison  with  actual  costs  incurred.  [Ref  37:pp.l8,  176] 
The  differences  must  be  explained  and  understood  for  the  model 
to  be  useful.  Without  that  understanding,  the  model  must  be 
modified  and  analyzed  until  it  does  produce  reliable  and 
accurate  estimates. 
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Calibration  of  a  model  is  basically  the  relation  of  a 
model  to  a  particular  instance  rather  than  a  general  set  of 
events.  This  would  be  a  consideration  in  cases  where  a 
company  uses  a  standard  commercial  version  of  a  software 
product  or  model  for  estimate  generation  rather  than  a  company 
that  developed  a  model  based  on  their  own  historical  data. 
[Ref  37:pp.l7,  162] 

8 .    Work  Breakdown  Structure 

In  the  Federal  Government  acquisition  process,  the  WBS  is 
often  used  as  a  base  upon  which  the  technical  requirements  for 
a  program,  system,  piece  of  equipment,  or  individual  contract 
action  are  developed.  [Ref  37:p.l77]  Through  a  hierarchical 
family  tree  structure  detailing  the  end  product,  work  elements 
to  achieve  production  of  the  end  product,  and  their 
relationships,  an  understanding  of  the  processes  involved  in 
the  accomplishment  of  an  objective  is  achieved.   Specifically: 


...the  work  breakdown  structure  plays  a  significant 

role  in  planning  and  assigning  management  and 

technical   responsibilities;   and  monitoring  and 

controlling   the   progress   and   status   of  (a) 

engineering  efforts,   (b) resource  allocations,  (c) 

cost  estimates,  (d)  expenditures,  and  (e)  cost  and 
technical  performance.  [Ref  37:p.l78] 


Guidance  regarding  the  use  of  the  WBS  is  provided  in  MIL-HDBK- 
881.  [Ref  37:p.l77]  DoD  policy  is  that  MIL-HDBK-881B  should 
be  referenced  for  guidance  only  in  appropriate  solicitations 
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defined  as  those  solicitations  involving  cost  risk  to  the 
Government.  The  intent  is  to  improve  program  and  contract  WBS 
development,  provide  guidance  to  the  contractor  in  extending 
the  program  WBS  and  maintain  the  integrity  of  the  WBS. 

9.    Price  and  Cost  Analysis 

Price  and  cost  analysis  are  used  to  determine  that  fair 
and  reasonable  prices  are  paid  for  products  in  business  as 
well  as  Federal  Government  procurement.  [Ref  2:pp.253,  262] 
Price  analysis  acts  to  keep  a  focus  on  the  value  of  an  item, 
and  parametric  cost  analysis  with  its  focus  on  cost  drivers  is 
a  clear  application  of  that  principle.  Price  analysis  allows 
a  buyer  to  compare  a  price  against  a  standard  as  a  yardstick 
to  determine  if  an  item  is  worth  the  cost.  Price  analysis 
techniques  are  generally  faster  and  do  not  cost  as  much  to 
perform  as  cost  analysis,  a  detailed  breakdown  of  costs  for 
each  cost  element  in  a  proposal,  but  price  analysis  cannot 
give  an  accurate  picture  of  what  will  be  paid  for  an  item  in 
all  cases.  Price  analysis  is  principally  performed  using 
these  four  methods  [Ref  2:p.253]: 

1.  An   analysis   of   competitive  price  proposals   is 
conducted. 

2.  A  comparison  with  regulated,  catalog,  or  market  prices 
is  performed. 

3.  A  comparison  with  historical  prices  is  performed. 

4.  An  independent  cost  estimate  is  used. 
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Price  analysis  only  provides  estimates  of  those  costs 
relative  to  the  accuracy  of  the  prediction.  It  does  not 
provide  exact  price  predictions  about  costs  that  will  be 
incurred  in  an  effort. 

Parametric  cost  estimating  ties  together  the  concepts  of 
price  analysis  and  cost  analysis.  It  is  one  of  the  methods  of 
comparing  a  price  in  the  present  with  those  historically  paid 
for  an  item  and  is  one  of  the  four  methods  a  buyer  has 
available  to  perform  price  analysis.  Parametric  cost 
estimating  combines  the  price  analysis  technique  of 
referencing  historical  prices  with  the  mathematical  analysis 
inherent  in  improvement  curve  theory  from  cost  analysis.  [Ref. 
2:p.267] 

10.   Cost  Realism 

Cost  realism  is  a  key  concept  in  the  source  selection 
process  entailing  understanding  a  cost  proposal  in  detail  and 
evaluating  the  accuracy  of  component  costs.  The  purpose  of 
cost  realism  is  to  give  a  source  selection  official  greater 
assurance  that  the  effort  will  cost  what  is  estimated  in  the 
proposal  when  the  work  is  complete.  [Ref  6: p.  3-1]  The 
inference  is  that  the  more  realistic  the  proposal  the  more 
likely  the  project  will  avoid  a  cost  overrun  situation. 
Specific  information  about  how  a  cost  proposal  will  be 
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evaluated  should  be  included  in  the  source  selection 
evaluation  plan  for  a  given  procurement  action  and  published 
in  applicable  solicitation  documents.  In  general,  the  Federal 
Government  will  assess  cost  realism  based  on  the  difference 
between  the  contract  offer  and  an  Independent  Government  Cost 
Estimate  (IGCE) .  Another  method  of  determining  cost  realism 
is  to  compare  the  proposed  costs  with  a  recalculation  of  the 
costs  by  the  buying  command  based  on  contractor-provided  data, 
past  performance,  and  information  from  DCAA  or  DCMC  audits. 
[Ref  6:p.ll-3] 
D.    PARAMETRIC  COST  ESTIMATING  APPLICATIONS  IN  ACQUISITION 

From  the  original  efforts  made  by  the  Rand  Corporation  to 
develop  a  method  of  accurately  estimating  costs  of  proposed 
new  systems,  the  uses  of  parametric  cost  estimating  have 
expanded.  Parametric  cost  estimates  have  been  and  continue  to 
be  used  to  [Ref  37:p.71] 

1.  Identify  possible  cost-performance  tradeoffs 

2.  Provide  a  basis  for  cost/effectiveness  review  of 
performance  specifications 

3.  Provide  information  useful   in  ranking  competing 
alternatives 

4.  Identify  and  consider  new  alternatives 

5.  Develop  target  cost  estimates  for  planning  purposes 

6.  Estimate  new  program  costs 
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7.  Make  decisions  during  bid  or  proposal  strategizing 

8.  Cross  check  actual  costs  during  program  life  cycle 
with  projected  cost  estimates  made  during  planning. 


All   of  these  uses  have  been  primarily  in  the  program 

management  area.   In  the  later  stages  of  a  program,  they  are 

used  in  a  secondary  role  to  bottoms  up  estimating.   In  fact, 

the  CPRG  has  the  use  of  CERs  in  price  analysis  shown  as  a 

secondary  comparison  technique.   Emphasis  is  now  on  using 

parametric  cost  estimates  as  a  primary  source  of  cost 

estimates  on  an  individual  contract  action  basis.   The  primary 

goal  is  to  maximize  efficiency  and  minimize  complexity  of  the 

solicitation,   the  evaluation,   and  the  selection  decision 

process . 

E.    ADVANTAGES  OF  AND  LIMITATIONS  IN  USING  PARAMETRIC  COST 
ESTIMATING  TECHNIQUES 

1 .    Advantages 

Parametric  pricing  using  CERs  has  the  advantages  of  being 
quick  and  simple  to  apply  once  the  relationship  has  been 
described  adequately.  [Ref  37:p.38]  This  is  a  result  of  the 
reduced  need  for  cost  data  specific  to  the  cost  elements  of 
the  new  item  or  system.  The  only  data  needed  are  the 
independent  variable  inputs  to  the  equations  to  get  an  output. 
Efforts  normally  used  to  generate  detailed  cost  estimates, 
that  may  be  no  more  accurate  than  the  parametric  estimates, 
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can  be  allocated  to  other  more  profitable  endeavors.  Where  a 
detailed  estimate  may  not  be  possible  because  of  the  lack  of 
data,  a  parametric  cost  estimate  may  be  possible. 

Parametric  cost  estimating  methods  do  not  have  to  be 
isolated  to  use  only  in  large  scale  applications  at  the 
program  or  even  entire  equipment  level.  [Ref  37:p.68]  They 
can  be  applied  at  the  component  level  for  inclusion  in  another 
type  of  cost  estimate  as  well  as  on  the  macro  management 
level.  They  provide  a  relatively  quick  and  accurate  way  to 
produce  cost  estimates  on  a  program  or  system  level  to  be  used 
as  a  sanity  check.  On  the  larger  scale,  when  used  to  estimate 
costs  at  the  system  or  program  level,  the  data  base  includes 
costs  incurred  as  a  result  of  setbacks  or  other  problems. 
With  the  database  built  on  a  continuing  series  of  changes, 
usually  in  high  technology  areas,  the  model  takes  advancing 
technology  automatically  into  account.  Thus  the  CERs  have  a 
built-in  correction  factor  for  problem  areas  often  observed  as 
well  as  for  program  cost  escalations  and  inflationary 
pressures . 

The  accuracy  of  parametric  cost  estimates  has  improved 
significantly  since  their  inception.  [Ref  25:p.5]  Using 
proven  databases  parametric  cost  estimating  models  are 
outstanding  predictors  of  accurate  estimates.  From  order  of 
magnitude   estimates   in   the   early   chemical   industry 


35 


applications,  to  being  equally  as  accurate  as  detailed 
engineering  cost  estimates  in  1975  [Ref  25:p.6],  the  accuracy 
of  parametric  cost  estimates  has  improved  to  the  present 
claims  of  five  percent  difference  from  actual  observed  costs 
in  some  uses  of  the  PRICE  models.  [Ref  21:p.l47]  As  in  any 
modeling  situation,  accuracy  of  the  inputs  is  a  critical 
factor  in  obtaining  useful  and  accurate  output  data. 
Parametric  cost  estimate  accuracy  is  highly  dependent  on  the 
following  variables  [Ref  14:p.l29]: 

1.  Degree  of  project  definition 

2.  Comparability  between  projects  in  the  database  and  the 
proposed  project 

3.  Status  of  model  calibration  and  validation 

4.  Data  availability  and  depth  of  the  database  used 

5.  Adjustments  made  to  the  data  to  ensure  comparability 
within  the  database 

Another  factor  to  consider  is  the  increasing  accuracy  levels 
achievable  throughout  the  life  of  a  project.  As  more  is  known 
about  a  project,  more  accurate  estimates  can  be  made  over 
time.  The  benefit  of  a  parametric  cost  estimating  system  is 
the  speed  at  which  revised  estimates  can  be  generated  as  new 
information  becomes  available.  The  flexibility  inherent  in 
the  parametric  cost  estimating  methods  is  also  demonstrated  in 
their  uses  in  conducting  "what  if"  analyses. 
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2.    Limitations 

Despite  all  of  the  advantages,  there  are  some 
difficulties  with  using  parametric  cost  estimating  methods. 
They  may  not  be  applicable  in  all  cases  because  one  may  not  be 
able  to  find  a  close  relationship  between  costs  and  any 
independent  variable  that  can  be  easily  measured  or 
quantified.  In  some  cases  the  models  use  CERs  that  do  not 
provide  sufficient  detail  to  accurately  predict  costs.  To 
maintain  accuracy,  the  historical  data  base  must  be  updated 
continuously,  or  the  equations  may  not  accurately  represent 
the  relationship  between  costs  and  the  independent  variable 
chosen  as  a  basis  for  estimation.  Technological  changes  in 
manufacturing  processes,  to  the  equipment  itself,  or 
significant  changes  in  the  management  structure  of  an 
organization  may  make  it  to  difficult  to  compare  and  correlate 
with  the  established  database.  [Ref  14:p.l30] 

Additionally,  several  problems  distort  the  results  from 
parametric  price  estimation.  Aside  from  inaccuracies  that  can 
be  introduced  by  design  into  the  equations  to  allow  greater 
applicability,  the  size  of  the  data  base  can  have  dramatic 
effects  on  the  accuracy  of  estimates.  [Ref  14:p.l32]  Larger 
data  bases  will  generally  have  greater  accuracies  than  smaller 
ones  as  the  statistical  models  increase  in  accuracy.  Data 
point  dispersion  will  also  affect  the  accuracy  but  this  can  be 
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measured  through  the  use  of  SSE  and  coefficient  of 
determination  values.  The  standardization  and  normalization 
of  data  that  is  required  to  be  able  to  compare  the  results 
across  programs  can  also  adversely  affect  the  output  of  a 
parametric  cost  estimating  model. 

Two  such  areas  of  normalization  are  in  the  adjustments 
for  inflation  by  Price  Level  Adjustments  and  Cost  to  Quantity 
Adjustments.  [Ref  16:p.l20]  The  data  to  be  analyzed  must  be 
adjusted  to  a  base  year  constant  dollar  level  for  comparison 
purposes  through  the  use  of  standard  indices.  To  compare 
across  different  procurement  sizes  in  terms  of  quantities  of 
units  purchased,  the  data  used  in  analysis  must  also  be 
adjusted  to  develop  consistent  cost  measures.  The  most  common 
method  used  to  do  this  is  through  the  use  of  learning  curve 
and  average  cost  per  unit  calculations  and  techniques.  [Ref 
37:p.l9] 

As  accurate  as  parametric  cost  estimating  is,  it  cannot 
give  a  100%  accurate  picture  of  what  will  be  paid  for  an  item. 
The  nature  of  estimating  is  to  predict  costs  in  the  future 
based  on  extrapolating  known  data  forward  in  a  new  situation. 
The  only  way  to  obtain  100%  accuracy  is  to  have  full 
visibility  of  costs  throughout  the  life  of  the  project  and 
total  them  at  project  completion. 
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Development  of  a  parametric  cost  estimating  model  and  its 
maintenance  is  expensive.  Database  management  itself  is 
costly  as  is  maintaining  a  system  of  data  collection  to  build 
and  update  the  information  required  to  keep  a  parametric  cost 
estimating  model  up  to  date.  Without  constant  maintenance, 
CERs  and  the  model  that  is  based  on  them  can  fall  out  of  date 
rapidly.  Finally,  although  parametric  cost  estimating  models 
are  very  flexible  within  the  boundaries  in  which  they  were 
derived,  the  degree  of  difficulty  in  redefining  those 
boundaries  makes  them  inflexible  in  accommodating  major 
program  changes . 
F .    SUMMARY 

Parametric  cost  estimating  has  enjoyed  a  long  history  and 
it  is  used  in  numerous  applications.  The  foundation  of 
parametric  estimating  system  is  the  integrity  of  the 
historical  database  and  the  CERs  that  make  up  the  model. 
Before  any  parametric  model  is  used,  the  model  and  database 
must  be  examined  to  ensure  that  they  are  up  to  date  and 
accurate.  The  model  should  be  calibrated  for  a  specific 
situation  and  then  validated  before  it  is  relied  upon.  With 
this  information  as  a  base,  the  following  chapter  will  use  the 
concepts  addressed  in  this  chapter  to  develop  a  cost 
estimating  model  using  parametric  methods. 
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III.   DEVELOPING  A  PARAMETRIC  COST  ESTIMATING  MODEL 

A.    PURPOSE 

This  chapter  will  describe  a  process  by  which  a 
parametric  cost  estimating  model  might  be  developed.  Knowing 
how  a  model  was  developed  allows  the  user  to  understand  the 
strengths  and  limitations  of  the  model.  This  in  turn  will 
lead  the  user  in  the  most  beneficial  application  of  the  model 
to  the  data  available  or  give  him  the  knowledge  he  will  need 
to  optimize  the  model  to  meet  his  specific  needs. 

There  are  eight  basic  steps  in  the  construction  of  a 
parametric  cost  estimating  model.  [Ref  25:pp.8-35] 


1 .  Gather  data  on  both  costs  and  potential  independent 
variables  such  as  speed,  weight,  or  number  of  lines  of 
computer  code . 

2.  Plot  the  data.  Through  a  simple  visual  analysis,  the 
user  may  determine  that  there  is  no  useful  correlation 
between  cost  and  the  independent  variable. 

3.  Mathematically  or  graphically  relate  the  data  points 
is  the  process  of  CER  formulation. 

4.  Examine  the  CER.  Does  the  CER  logically  apply  to  the 
data? 

5.  Refine  the  CER  and  review  the  data. 

6.  Test  the  CER.  Does  the  CER  predict  costs  comparably 
to  known  historical  costs  for  the  variables  used  in  the 
CER? 

7.  Combine  the  CERs  in  the  Parametric  Cost  Estimating 
Model . 
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8.  Test  the  Parametric  Cost  Estimating  Model.   Does  the 
model  predict  costs  comparably  to  known  historical  costs? 

For  the  purposes  of  this  effort  in  model  construction,  a  study 

by  Raymond  E.  Moore  III  in  1975,  Cost  Estimating  Relationships 

For  Naval  Surface  Ship  Electronic  Warfare  Equipment,  will  form 

the  basis.   [Ref  25]    In  that  study,  Moore  developed  a 

parametric  cost  estimating  model  for  electronic  equipment 

based  on  a  review  of  the  costs  and  technical  data  for  12 

existing  systems.   These  data  are  shown  in  Table  1. 

Table  1 
Electronic  Warfare  Equipment  Data  Matrix 


System 

Unit 
Cost 

Year 

Developm 
ent  Cost 

Instal 
lation 
Cost 

Weight 

Volume 

Sensitiv 
ity 

Power 
Usage 

Quantity 

Gain 

SLQ-19 

758 

69 

1886 

268.1 

5190 

275.2 

-35 

15.0 

4 

0 

SLQ-26 

1572 

70 

3654 

582.5 

6016 

262.1 

-36 

15.0 

8 

15 

SLQ-28 

4092 

71 

11694 

562.5 

21032 

560.7 

-20 

15.0 

5 

15 

WLR-11 

120 

72 

360 

245.3 

375 

16 

-65 

12 

0 

SLQ-30 

408 

74 

1600 

37.0 

2060 

131.6 

-45 

1.0 

100 

15 

SLQ-12 

250 

65 

1754 

52.1 

2614 

154.2 

-50 

.4 

50 

20 

SLQ-17 

1625 

73 

7256 

168.9 

6000 

450 

-48 

15.0 

30 

15 

SLR- 12 

6.8 

65 

380 

15 

200 

12 

-60 

315 

18 

WLR-1 

30.0 

63 

3598 

280.1 

1958 

185 

-80 

600 

15 

WLR-3 

3.0 

63 

45 

25 

1 

-45 

400 

15 

WLR-8 

1200 

73 

11694 

2908 

220 

-85 

7 

15 

Source:  Reference  24,  p.  40 
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B. 


PLOTTING  DATA 


Moore  compiled  these  data  and  analyzed  the  relationships 
between  both  development  and  installation  costs  and  all  of  the 
parameters  listed  in  Table  1.  [Ref  25:pp. 11-21]  For 
illustration  purposes,  this  study  will  demonstrate  the  second 
step  in  the  development  cycle  by  plotting  the  system  weight 
against  the  development  costs  in  Figure  2. 

Electronic  Equipment 

Weight  Vs.  Cost 
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Figure  2:  Weight/Cost  Data  Plotted  Graphically 

Source:  Developed  by  researcher. 

At  this  point,  a  visual  inspection  shows  that  although 
there  are  some  outlying  data  points,  it  appears  as  if  a  useful 
mathematical  relationship  can  be  derived  from  the  points.  In 
contrast,  Figure  3  is  a  graph  of  system  development  costs 
plotted  against  quantities  of  the  systems  acquired.  From 
visual  inspection  of  the  graph  there  appears  to  be  no  useful 
correlation  between  these  two  variables. 
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Electronic  Equipment 

Quantity  Vs.  Cost 
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Figure  3:  Quantity/Cost  Plotted  Graphically 

Source:  Developed  by  researcher. 

To  develop  a  CER  to  be  used  in  parametric  estimation,  the 
buyer  has  to  first  decide  what  he  wants  to  estimate  (cost, 
schedule,  material,  labor,  or  some  other  parameter)  and  then 
determine  how  the  variables  are  related  in  the  work  to  be 
performed.  Will  the  price  estimate  be  derived  directly  from 
the  data  or  tied  to  a  variable  to  which  a  cost  is  then 
applied?  Here,  it  has  been  determined  that  cost  will  be  the 
dependent  variable  to  be  derived  directly  from  the  independent 
variable,  weight. 
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C.    CER  DEVELOPMENT 

Having  selected  the  characteristic  that  will  give  an 
accurate  representation  of  the  relationship  between  the 
independent  and  dependent  variables,  the  relationship  must  be 
quantified.  The  independent  and  dependent  variables  must  be 
not  only  related  in  some  manner,  but  they  must  be  related  in 
a  definable  way.  There  has  to  be  an  understandable 
relationship  between  the  data  points,  and  the  data  points  must 
be  easily  measurable  or  inferred  from  available  data. 

Using  linear  regression  techniques  a  line  can  be  used  to 
approximate  the  relationship  between  the  data  points.  Here 
the  relationship  between  development  cost  and  weight  can  be 
described  as  Y=  (.950082  x  System  Weight)  +  604.4864.  This 
CER  is  shown  in  Figure  4. 


Electronic  Equipment 


Weight  Vs.  Cost 
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Figure  4:  Cost/Weight  CER  Overlaid  on  Data  Graph 


Source:  Developed  by  researcher 
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The  coefficient  of  determination  for  this  CER  is 
calculated  to  be  .691395  and  the  SSE  to  be  4519.926.  A 
coefficient  of  determination  with  a  value  of  1.0  indicates  a 
perfect  correlation  between  the  two  variables.  In  this  case, 
for  illustration  purposes,  the  coefficient  of  determination  is 
close  enough  to  a  value  of  1  to  be  useful  as  an  indicator  of 
future  costs.  The  SSE  is  relatively  large,  in  part  due  to  the 
wide  range  of  development  cost  values  used  in  the  computation 
and  the  small  database  used. 
D.    COST  ESTIMATING  RELATIONSHIP  REVIEW/REFINEMENT 

A  test  of  the  CER  developed  here  shows  that  it  does  make 
intuitive  sense.  As  one  would  expect,  costs  appear  to  rise  as 
the  weight  of  the  system  increases.  An  examination  of  the 
relationship  and  the  data  supporting  it  are  now  made  in  an 
attempt  to  increase  the  accuracy  of  the  CER.  Should  some  of 
the  outlying  data  points  be  discarded  and  is  the  mathematics 
used  logically  sound  are  two  questions  that  have  to  be 
answered  each  time  the  CER  is  used.  As  an  illustration  to 
show  the  effects  of  data  manipulation,  two  data  points  will  be 
discarded  and  the  results  shown  graphically  in  Figure  5. 

Deleting  the  two  data  points  from  the  database  has  a 
great  effect  on  the  interpretation  of  the  information.  The 
CER  developed  has  changed  to  Y=  .834494  (Weight)  +  1300.296 
with  a  coefficient  of  determination  of  .777718  and  a  SSE  of 
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1643.74.  While  this  action  has  increased  the  accuracy  of  the 
CER,  it  may  have  also  reduced  the  effective  range  over  which 
it  is  valid.  Any  use  of  the  two  CERs  developed  must  take 
these  data  manipulations  into  account. 
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Figure  5:  Revised  Data  and  CER 

Source:  Developed  by  researcher. 

E.    COMBINING  CERS  INTO  PARAMETRIC  COST  ESTIMATING  MODELS 

Concluding  the  data  manipulation  and  CER  refinement 
initiates  the  final  step  in  the  process  of  CER  development. 
In  this  construction,  historical  costs  are  compared  with  costs 
that  would  be  projected  by  the  CER  using  the  known  weights  for 
the  system.  Table  2  presents  those  data  for  review.  Here,  it 
will  be  assumed  that  the  CER  data  compares  favorably  with  the 
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historical  data  and  that  they  fall  within  an  acceptable  range 

for  estimating  costs. 

Table  2 
Actual  vs.  CER  Projected  Costs 


|  System 


Weight 


Development 
Cost 


CER  Projected 
Cost 


!  SLQ-19 


5190 


1886 


5535 


SLQ-26 


6016 


3654 


6320 


WLR-1 1 


375 


360 


961 


i  SLQ-30 


2060 


1600 


2562 


i  SLQ-12 


2614 


1754 


3088 


|  SLQ-17 


6000 


7256 


6305 


!  SLR-12 


200 


380 


795 


!  WLR-1 


1958 


3598 


2465 


!  WLR-3 


25 


45 


628 


Source:  Developed  by  researcher. 

Up  to  this  point,  the  construction  has  been  concerned 
with  developing  a  single  CER  for  estimating  development  costs. 
The  purpose  of  the  effort,  however,  is  to  derive  a  parametric 
cost  estimating  model  for  a  new  electronic  system. 
Development  costs  comprise  only  a  part  of  the  costs  of  any 
system.  Other  costs  that  might  be  considered  are  installation 
costs,  spare  parts,  and  training  for  operators.  To  build  a 
parametric  cost  estimating  model,  all  of  these  factors  must  be 
combined  in  some  form.  They  may  be  estimated  using  a  CER,  as 
were  the  development  costs,  or  by  any  of  the  other  cost 
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estimating  techniques  available.  A  parametric  cost  estimating 
model  considering  these  and  other  factors  developed  by  Moore 
for  this  data  set  can  be  described  by  the  following  equation 
[Ref  23:p.21] : 


PROC  =  2811.59351  +  .05553  (DEVC)  +  .02824  (Weight)  +  3.81016 
(Volume)  +  25.98703  (Sensitivity)  +  73.68718  (Power  Usage)  + 
77.15399  (Gain)  -  1667.51343  (ACT)  -  1027.33765  (LG)  - 
1718.32324  (SM) 


which  had  the  following  statistics: 
Coefficient  of  Determination=  .9997 
Standard  Error  of  the  Estimate  =65.119 


where : 

PROC  -  Procurement  Cost  DEVC  -  Development  Cost 

ACT  -  Active  Equipment  Code        LG  -  Large  Vessel  Code 
SM  -  Small  Vessel  Code 

With  the  parametric  cost  estimating  model  complete,  there 
is  only  testing  against  historical  costs  to  verify  the 
accuracy  of  the  model  remaining.  The  process  described  above 
is  illustrative  of  one  method  available  to  develop  a 
parametric  cost  estimating  model.  Using  the  parametric  cost 
estimating  model  developed,  the  methods  can  be  used  in  any  of 
a  program's  life  cycle  stages  for  any  number  of  estimating 
purposes . 
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F.    SUMMARY 

This  chapter  developed  a  CER  using  regression  analysis  to 
demonstrate  the  relationship  between  the  database  used  and  the 
basis  for  the  estimate.  From  this  and  other  CERs  particular 
to  the  equipment  that  is  to  be  developed,  a  model  would  be 
constructed.  This  model  would  then  be  calibrated  if  that 
action  was  required  and  finally  validated.  After  that  point, 
it  could  be  used  to  estimate  the  costs  of  electronic  systems 
of  a  similar  nature  to  the  ones  that  made  up  the  database. 
The  next  chapter  describes  the  data  collection  and  analysis  of 
the  data  from  surveys  and  interviews. 
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IV.  METHODOLOGY  AND  DEMOGRAPHICS 

A.  PURPOSE 

This  chapter  will  discuss  the  data  collection  methods 
used  in  the  study.  Construction  of  the  survey  vehicle, 
Appendix  C,  and  interview  questionnaire,  Appendix  D,  will  be 
described  followed  by  a  discussion  of  the  methods  used  to 
disseminate  the  survey.  Data  in  this  chapter  will  be 
presented  as  aggregate  information  describing  the  demographic 
breakdown  of  the  respondents. 

B.  DATA  COLLECTION 

Data  collection  was  centered  around  the  -  primary  and 
secondary  research  questions.  Information  regarding  the  area 
of  interest  was  collected  throughout  the  use  of  a  self- 
administering  survey  and  personal  interviews.  The  research 
questions  formed  the  backbone  of  the  survey,  breaking  the 
survey  into  discrete  areas  of  interest.  The  survey  was 
structured  to  follow  a  possible  pattern  of  use  for  parametric 
cost  estimating  techniques  through  an  acquisition  cycle.  Each 
major  question  is  broken  into  several  subsets  to  allow  greater 
analysis  of  the  topic  in  question. 

Questions  one  through  four  form  the  basis  of  the 
discussion  of  the  use  of  parametric  cost  estimating  in  the 
procurement  process.   Ascertaining  the  risks  inherent  in  the 
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program,  system  or  equipment,  its  similarity  to  existing 
efforts,  and  the  development  of  the  databases  available  for 
analysis  are  key  elements  in  determining  how  parametric  cost 
estimating  can  be  used  in  acquisition  planning.  Questions  one 
through  four  represent  some  of  the  management  areas  to  which 
parametric  cost  estimating  have  been  and  can  be  applied  within 
the  acquisition  cycle. 

From  the  basis  formed  by  questions  one  through  four,  the 
survey  moves  on  to  obtain  opinions  regarding  the  subsidiary 
research  question:  What  changes  will  offerors  make  to  their 
proposal  preparation,  submission,  and  support  processes  to 
accommodate  parametric  methods  in  estimating  contract  costs? 
Question  five  transitions  from  the  general  uses  of  parametric 
cost  estimating  methods  to  issues  specific  to  the  conduct  of 
pre-award  contract  actions.  Question  seven  continues  to 
gather  opinions  regarding  the  question  of  offeror  proposal 
preparation.  The  buyer's  estimation  of  his  position  regarding 
the  purchase  of  an  item  will  be  based  on  his  understanding  of 
the  risks  in  the  acquisition  cycle  and  will  be  reflected  in 
the  solicitation  he  develops. 

The  structure  of  a  procurement  action  is  in  large  part 
affected  by  its  place  in  the  acquisition  cycle  of  a  program 
and  the  overall  acquisition  strategy.  In  turn,  an  offeror's 
proposal  is  driven  by  the  solicitation  to  which  it  is 
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responding.  Elements  of  question  six  and  question  eight 
address  the  subsidiary  research  question:  What  impacts  will 
emphasis  on  parametric  cost  estimates  have  on  source  selection 
criteria  and  on  the  evaluation  of  proposals  by  the  DoD?  The 
remaining  elements  of  question  six  and  question  nine  induce 
the  survey  respondent  to  consider  the  manner  in  which  a 
solicitation  is  conducted. 

A  central  tenet  in  the  Federal  Government  acquisition 
process  is  the  fairness  of  the  process.  These  questions 
collect  data  relating  to  the  subsidiary  research  question:  How 
will  source  selection  criteria  and  evaluation  factors  be 
constructed  to  fairly  evaluate  a  proposal  that  uses  a 
parametric  cost  estimate  with  one  that  does  not?  The  final 
area  addressed  in  the  survey  is  the  negotiation  process. 
Question  ten  collects  opinions  to  answer  the  question:  What 
effects  will  using  parametric  cost  estimating  have  on 
negotiations?  An  additional  page  was  attached  for  comments 
regarding  the  questions  or  any  other  concerns  expressed  by  the 
respondents.  These  additional  data  have  been  incorporated 
into  the  analysis  where  applicable. 

In  each  of  the  questions  in  the  survey,  the  respondents 
are  asked  to  record  their  opinions  about  the  use  of  parametric 
cost  estimating  methods  using  a  numerical  scale.  Five  point 
numerical  scales  were  selected  to  provide  a  consistent,  simple 
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method  of  coding  the  responses  for  data  reduction  and 
analysis.  Two  different  scales  were  used  in  the  survey. 
Questions  one  through  four  used  a  scale  describing  the  utility 
of  parametric  cost  estimating  methods  and,  questions  five 
through  ten  used  a  scale  describing  agreement  with  the 
statements  that  make  up  the  elements  of  those  questions. 
Demographic  data  were  also  collected  in  an  attempt  to  describe 
the  population  of  respondents.  The  data  were  examined  for 
relationships  between  subgroups  represented  in  the  survey 
respondent  population  and  responses  for  particular  questions. 
Through  these  methods  the  data  were  reviewed,  analyzed,  and 
related  to  the  research  questions. 

An  interview  questionnaire,  Appendix  D,  was  developed  in 
the  same  manner  as  the  survey  instrument.  The  basic  and 
subsidiary  research  questions  were  addressed  in  the  interview 
questionnaire.  The  interview  questionnaire  was  presented  to 
seven  individuals  in  personal  interviews  at  the  Naval 
Postgraduate  School  over  the  course  of  the  research. 
Additionally,  an  initial  form  of  the  interview  questionnaire 
was  given  to  six  people  via  telephone  and  E-mail  at  the  start 
of  the  research  effort.  The  information  provided  has  been 
inserted  in  the  discussion  and  background  areas  preceding  this 
chapter  as  well  as  in  the  analysis  of  the  survey  data  where 
applicable. 
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The  survey  was  distributed  as  widely  as  possible  to 
attempt  to  include  as  many  elements  of  the  DoD  acquisition 
community  as  possible.  The  bulk  of  the  surveys  were 
distributed  by  mail  using  Federal  Government  agency  mailing 
lists  and  organizational  information.  Electronic  mail  and 
facsimile  transmissions  were  also  used  to  distribute 
approximately  25%  of  the  surveys.  Potential  survey 
participants  were  also  identified  through  INTERNET  searches 
and  referrals  during  interviews.  Through  the  survey's  wide 
distribution  across  the  DoD  acquisition  community,  an 
understanding  of  the  potential  for  the  uses  of  parametric  cost 
estimating  methods  was  generated.  Appendix  E  is.  a  collection 
of  the  data  derived  from  the  survey  effort. 

An  important  group  solicited  for  input  were  the  members 
of  the  Federal  Government's  Parametric  Estimating  Initiative 
Steering  Committee.  This  group  was  identified  following  the 
preface  in  Reference  36  and  represents  a  cross  section  of  the 
DoD  acquisition  community.  The  Steering  Committee  came 
together  to  identify  methods  of  promoting  the  uses  of 
parametric  cost  estimating  techniques  throughout  the  DoD 
community.  For  purposes  of  this  survey,  they  responded  to  a 
pilot  survey  with  recommendations  and  suggestions  for 
improvement  to  the  survey  structure,  direction,  and  the 
wording  of  questions.   Their  answers  to  the  questions  were 
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also  most  valuable  in  that  they  formed  a  basis  for  comparison 
against  which  the  other  demographic  groups  could  be  compared. 
Resident  in  this  group  is  the  most  current  information  about 
the  uses  of  parametric  cost  estimating  across  the  entire 
spectrum  of  actions  inherent  in  the  DoD  acquisition  process. 
The  Steering  Committee  is  referred  to  as  the  control  group 
throughout  the  analysis  portion  of  this  chapter. 
C.    DEMOGRAPHIC  DATA 

The  survey  distribution  was  tracked  through  the  employer 
question  of  the  survey.  The  industry  and  Federal  Government 
employer  categories  were  all-encompassing,  mutually  exclusive 
groupings  that  were  easily  tracked.  Using  this  breakdown  as 
a  basis,  1260  surveys  were  distributed  over  the  period  21 
August  1996  to  15  October  1996.  Throughout  the  research 
effort,  it  was  found  that  many  addresses  were  incorrect.  It 
was  assumed  that  the  incorrect  addresses  resulted  from  base 
closures  and  realignments  or  agency  reorganization  efforts. 
Where  possible,  correct  addresses  were  obtained  and  the  survey 
redistributed.  Figure  6  represents  the  survey  distribution 
throughout  the  data  collection  effort. 
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Survey  Distribution 

Employer  Groups 


DCMC  (14.45%) 
Industry  (2.21%) 

DOE  (3.91%) 

OSD  (0.78%) 
NASA  (2.78%) 

USN  (8.54%) 
GSA  (0.64%) 

USA  (23.63%) 


DCAA  (37.37%) 


USAF  (5.69%) 


Figure  6:  Research  Survey  Distribution 

Source:  Developed  by  researcher. 

The  majority  of  the  surveys  were  sent  .  to  Federal 
Government  contract  support  activities,  DCMC  and  DCAA  (52%)  . 
It  was  felt  that  each  organization  interacting  with  a  major 
portion  of  the  DoD  acquisition  community  across  the  entire 
spectrum  of  procurement  actions  would  represent  the  DoD 
community  experience  with  the  uses  of  parametric  cost 
estimating  methods. 

A  low  survey  return  rate  was  observed  from  the  data 
collected  from  210  surveys  returned.  Only  16.5%  of  the 
surveys  distributed  were  returned  with  data  to  be  input  for 
analysis.  No  opinion  or  no  response  to  one  or  more  questions 
was  recorded  on  47%  of  the  surveys  returned.  Figure  7  shows 
the  distribution  of  the  returned  surveys. 
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Group  Return  Rates 


Industry  - 
OSD 

DoE-ii 


GSA- 1 

USAF  - 


U.S.  Navy  -\ 
NASA  -i 
DCAA-: 


DCMC -U~ 


U.S.  Army  -i 


0        10       20       30       40       50       60       70 


%  Surveys  Returned 


Returned  Surveys 


DCMC  16.83% 


DCAA  23.08% 


U.S.  Army  24.99% 


Industry  9.61% 


NASA  4.33% 

U.S.  Navy  10.58% 


OSD  0.96% 
DoE  0.96% 
GSA  0.48% 
USAF  8.18% 


Figure  7 :  Survey  Return  Rates 
Source:  Developed  By  Researcher 
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The  percentages  in  the  Returned  Surveys  chart  of  Figure 
7  display  the  percent  of  the  surveys  returned  from  an  employer 
group.  The  Group  Return  Rate  chart  of  Figure  7  shows  the 
percentage  of  the  surveys  sent  to  each  group  that  were 
returned  by  members  of  those  targeted  employer  groups.  The 
percentages  of  the  surveys  returned  by  groups  ranged  between 
68%  from  industry  participants  to  a  low  of  3.64%  by  Department 
of  Energy  (DoE)  personnel,  both  small  groups  within  the  survey 
population.  Despite  the  relatively  small  number  of  the 
industry  participants  targeted  (2%) ,  their  high  return  rate 
made  them  the  fifth  largest  group  contributing  to  the 
database.  The  industry  group  may  have  had  a  disproportionate 
affect  on  the  response  averages  because  of  this  high  return 
rate.  The  remaining  groups  had  fairly  similar  return  rates 
within  a  much  smaller  range  of  12%  to  25%  and  formed  the 
majority  of  the  data  reviewed. 

As  the  survey  was  designed  to  ensure  anonymity  of  the 
respondent,  no  effort  was  made  to  track  response  rates  by 
organization  within  an  agency,  individually,  or  by  any  other 
demographic  breakdown  other  than  that  shown  above.  The 
researcher  was  contacted  on  an  individual  basis,  however,  by 
approximately  three  percent  of  the  survey  population  who 
indicated  that  they  would  not  be  participating.  All  of  these 
respondents  indicated  that  they  were  unfamiliar  with  the  topic 
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and  did  not  feel  qualified  to  join  in  the  survey.  While 
parametric  cost  estimating  methods  have  been  in  use  within  DoD 
for  many  years,  their  use  has  typically  been  isolated  to  upper 
and  middle  management  areas  dealing  with  major  programs.  Their 
use  has  not  been  as  widespread  in  the  other  areas  of  DoD 
procurement  that  make  up  the  bulk  of  the  procurement 
transactions.  [Ref  45:pp.7-ll]  It  is  likely  that  a  lack  of 
experience  with  parametric  cost  estimating  methods  is  a  major 
factor  in  the  survey's  low  return  rate. 

The  timing  of  the  survey  may  also  have  had  a  negative 
impact  on  the  return  rate  observed.  The  survey  was 
distributed  at  the  end  of  the  Federal  Government's  fiscal 
year,  typically  one  of  the  most  active  times  of  the  year  in 
the  DoD  acquisition  community.  Although  the  length  of  the 
survey  was  minimized  to  encourage  participation,  the  survey 
had  55  individual  data  elements  to  be  considered.  Many  of  the 
recipients  may  not  have  had  time  to  complete  the  survey  within 
the  requested  time  frame  and  thus  chose  not  to  participate. 

Demographic  data  were  requested  in  four  areas  beyond  the 
employer  classification  previously  discussed.  Participants 
were  asked  to  identify  themselves  by  their  job  classification, 
the  length  of  time  that  they  had  been  working  in  this  area, 
how  they  categorized  their  work,  and  their  level  of  education 
in  mathematics.    Each  question  was  examined  with  these 
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categories  in  mind  to  identify  trends,  correlations,  or  points 
of  interest  in  the  data  collected.  In  terms  of  job 
classification,  three  categories  comprised  the  majority  of  the 
survey  respondents:  those  identifying  themselves  as  Cost  or 
Price  Analysts  (26%),  Contracting  Officers  (23%),  and  Auditors 
(22%) .  Small  numbers  of  participants  identified  themselves  in 
the  remaining  job  classification  groups. 

Two  employer  groups  were  very  homogeneous  in  composition: 
the  DCMC  and  DCAA  participants.  The  DCMC  group  identified 
themselves  almost  entirely  within  the  job  classification  of 
contracting  officer.  The  DCMC  personnel  made  up  70%  of  the 
responding  individuals  of  those  identifying  themselves  as 
contracting  officers.  Unlike  the  DCMC  participants,  the  DCAA 
group  makes  up  almost  all  of  the  auditor  job  classification 
respondents.  The  DCAA  and  Auditor  groups  are  nearly 
synonymous,  with  96%  of  Auditors  also  classifying  themselves 
as  working  for  DCAA.  All  of  the  other  groups  were  made  up  of 
combinations  of  the  job  classifications. 

The  majority,  49%,  of  the  participants  identified 
themselves  within  the  acquisition  community  and  only  19%  in 
the  cost  estimating  community.  A  large  portion  of  the 
respondents  did  not  chose  to  respond  to  this  question,  perhaps 
overlooking  it.  There  was  no  observed  correlation  between 
either  community  and  any  other  demographic  category  similar  to 
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that  observed  in  the  auditor  and  contracting  officer  job 
classifications.  The  majority  of  the  respondents  were  very 
experienced  in  their  positions,  having  more  than  10  years  of 
experience  in  either  the  acquisition  or  cost  estimating 
professions.  Very  few  people,  11%  of  the  210  participants, 
had  less  than  10  years  experience.  Also  of  note,  only  39%  of 
the  respondents  chose  to  provide  their  GS/SES  level  or 
military  rank.  The  resulting  analyses  from  these  groups  are 
limited  because  of  their  small  size. 

The  self-described  work  classifications  are  unlike  the 
other  demographic  breakdowns  in  that  these  groups  are  not 
mutually  exclusive.  In  question  four,  respondents  were  asked 
to  select  any  of  the  work  functions  listed  that  they 
performed.  They  were  not  to  limit  their  response  to  any 
single  work  function  that  made  up  the  majority  of  their  work 
effort.  Reviewing  or  auditing  estimates  was  chosen  to 
describe  the  work  most  often  performed  by  the  respondents;  103 
selected  this  work  description.  This  group's  answers  roughly 
correspond  to  those  in  the  job  classifications  of  cost  and 
price  analyst  and  auditor.  Negotiation  preparation  and 
proposal  analysis  was  also  selected  by  a  large  number  of 
participants,  101  of  210,  to  describe  their  work  efforts. 
Here,  there  is  a  correlation  between  this  group  of  responses 
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and  those  of  the  contracting  officer,  contract  negotiator,  and 
auditor  job  classifications. 

Within  the  database,  the  majority,  88%,  of  the 
respondents  answered  that  they  had  some  degree  of  college 
mathematics  education.  A  significant  number  of  the 
participants  had  either  college  level  algebra  or  business 
calculus,  30%  and  33%  of  the  respondents  respectively.  Of 
those  identifying  themselves  as  having  taken  an  engineering 
calculus  series,  only  30%  also  identified  themselves  as 
engineers.  The  remaining  individuals  identified  were  spread 
across  the  rest  of  the  job  classifications  as  were  the  people 
who  answered  that  they  had  either  advanced  degrees  or  majored 
or  minored  in  mathematics  while  a  college  undergraduate. 

After  reviewing  the  basic  statistics  for  the  entire 
database,  the  database  was  subdivided  and  analyzed.  The 
information  presented  above  and  the  demographic  description  in 
Appendix  E  were  used  to  break  the  database  into  separate 
components  to  be  compared  against  each  other  and  the  entire 
database  statistics.  In  the  following  Chapter  the  comparisons 
made  and  analyses  performed  on  the  data  will  be  described. 
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V.  APPLICATIONS  IN  PROGRAM  MANAGEMENT  FUNCTIONS 

A.    PURPOSE 

This  chapter  and  the  following  two  chapters  will  discuss 
the  data  analysis  methods  used  and  results  observed  in  the 
study.  Chapter  V  analyzes  results  from  question  one  data 
related  to  parametric  cost  estimating  applications  in  the 
broad  scope  of  program  management  functions.  Data  pertinent 
to  survey  question  one  will  be  presented,  related  to  the 
research  question,  and  analyzed.  Questions  one  through  four 
used  a  five  point  scale  that  ranged  from  "vital  to  the 
procurement  process,"  a  one  on  the  scale,  to  "a- hindrance  to 
the  procurement  process,"  a  five  on  the  scale.  Each  question 
was  further  broken  into  smaller  areas  of  specific  interest. 

The  data  from  each  question  were  analyzed  and  the 
following  statistics  derived: 

1.  The  number   of   responses   in  each  of   the   five 
categories . 

2.  An  average  value  of  the  responses. 

3.  The  standard  deviation  for  the  responses. 

4.  The  variation  of  the  responses. 

5.  The  median  value  of  the  responses. 

6.  The  skewness  of  the  responses. 
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These  data  are  presented  in  Appendix  E  for  the  entire  group  of 
responses  as  well  as  for  each  demographic  group. 

Analysis  of  all  the  questions  and  their  components  was 
conducted  in  a  similar  manner.  Descriptive  statistics  for  the 
entire  group  are  presented,  followed  by  group  breakdown 
information,  and  the  data  presentation  closes  with  analysis  of 
variance  and  regression  information.  These  data  are  presented 
in  tables  following  an  introductory  paragraph.  Group  data  for 
each  question  was  chosen  to  be  displayed  for  one  of  three 
reasons : 


1 .  Some  degree  of  consensus  was  observed  in  the  group 
data  analysis 

2.  The  group  appeared  as  a  point  of  interest  in  the 
analysis  of  variance,  or 

3.  The  group  data  are  illustrative  of  a  point  of  interest 
found  during  the  regression  analysis  performed. 


In  both  the  regression  analyses  and  analyses  of  variance,  a 
95%  confidence  interval  was  used.  For  simplicity,  group 
descriptive  statistics  for  the  first  subgroup  in  the  first 
question,  describing  the  utility  of  parametric  cost  estimating 
in  the  area  of  benchmarking  process  costs,  will  be  presented 
in  tabular  and  text  format  with  the  data  for  the  remaining 
questions  shown  in  tabular  format  only.  An  analysis  of  the 
displayed  data  will  follow  those  data  for  each  question. 
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B.    BROAD  SCOPE  UTILITY 

The  first  question  asked  the  respondent  to  provide  an 
opinion  regarding  the  utility  of  parametric  cost  estimating 
methods  in  several  functional  areas.  This  question  was 
divided  into  12  segments,  five  dealing  with  .  overarching 
functions  common  to  the  acquisition  cycle  in  general  and  seven 
treating  procurement  of  specific  items  or  services. 

1 .    Benchmarking  Process  Costs 

Benchmarking  process  cost  was  the  initial  subgroup  within 
question  1 .  Parametric  cost  estimating  methods  may  have 
applications  in  situations  where  new  processes  are  being 
developed.  The  methods  may  be  most  useful  •  where  these 
processes  are  sufficiently  similar  to  past  or  existing 
processes  and  the  new  processes  are  expected  to  have  similar 
cost  characteristics  or  patterns.  As  a  group,  the 
respondents'  opinions  indicated  that  they  thought  parametric 
cost  estimating  methods  were  somewhat  useful  in  this 
functional  area.  The  largest  single  group,  56  respondents, 
selected  somewhat  useful,  a  three  on  the  scale,  as  describing 
the  methods'  degree  of  utility  here.  Only  one  respondent 
thought  that  the  methods  could  hinder  cost  estimating  if  used 
here  and  21  people  thought  they  were  of  no  use. 

The  corporate  policy  group  selected  either  vital  or  very 
useful  to  describe  the  utility  of  parametric  cost  estimating 
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methods  in  this  situation.  This  agreement  is  statistically 
demonstrated  in  the  small  standard  deviation  of  .548  for  the 
group  and  a  mean  value  of  1.4  for  all  of  the  responses.  These 
statistics  must  be  tempered,  however,  with  the  group  size. 
With  only  five  individuals,  it  is  a  very  small,  part  of  the 
database  of  210.  Other  small  groups  had  similar  occurrences 
in  the  data  analysis.  The  program  staff  group,  five  persons, 
three  industry  cost  analysts  and  the  U.S.  Army  cost  analyst 
group,  a  group  of  nine,  were  unanimous  in  their  belief  that 
parametric  cost  estimating  methods  were  somewhat  useful  for 
benchmarking  process  costs.  U.S.  Air  Force  cost  analysts, 
another  small  group  of  five,  agreed  with  the  cost  analyst  from 
the  General  Services  Administration  (GSA)  that  the  methods 
were  very  useful  for  this  purpose.  These  data  described  are 
displayed  in  Table  3.  Analyses  of  variance  and  regression 
analyses  pertinent  to  question  one  and  discussions  of  these 
results  follow.  No  other  trends  or  indications  were  observed 
from  the  group  statistics  analyzed. 

Other  analyses  performed  with  the  data  from  this  question 
were  analysis  of  variance  and  regression  analysis.  In 
reviewing  the  results  shown  in  Table  4,  a  relationship  was 
noted  between  job  classification  and  the  responses  for 
benchmarking  process  costs.  A  p  value  of  .04  indicates  that 
a  relationship  exists  between  the  responses  in  the  job 
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classification  group  and  their  answers   for  benchmarking 

process  costs. 

Table  3 
Benchmarking  Process  Costs  Group  Data 


Group 
Description 

Min 

Max 

M 

Median 

a 

Ql 

Q3 

Corporate 
Policy 

1.00 

2.00 

1.40 

1.00 

.55 

1.00 

2.00 

Program 
Staff 

3.00 

3.00 

3.00 

3.00 

.00 

3.00 

3.00 

Industry 

Cost 

Analysts 

3.00 

3.00 

3.00 

3.00 

.00 

3.00 

3.00 

USA  Cost 
Analysts 

3.00 

3.00 

3.00 

3.00 

.00 

3.00 

3.00 

USAF  Cost 
Analysts 

2.00 

2.00 

2.00 

2.00 

.00 

.2.00 

2.00 

GSA  Cost 
Analysts 

3.00 

3.00 

3.00 

3.00 

.00 

3.00 

3.00 

Source:  Developed  by  researcher. 

A  point  of  interest  in  this  analysis  is  in  the  difference 
of  opinion  between  the  program  managers  and  other  job 
classification  groups.  The  six  program  managers  who  chose  to 
respond  to  this  question  believed  that  the  methods  were  vital 
or  very  useful  in  benchmarking  process  costs.  The  other 
groups  felt  that  the  methods  were  either  very  useful  or  only 
somewhat  useful  for  this  application.  Analysis  of  variance 
between  the  question  responses  and  the  DoD  employer  categories 
also  showed  that  a  relationship  exists.  The  majority  of  the 
participants  (87),  represented  by  the  U.S.  Army,  DCMC,  and 
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DCAA,  responded  that  the  methods  were  somewhat  useful  where 
the  remaining  respondents  (31)  answered  that  the  methods  were 
very  useful  to  the  process. 


Table  4 
Analysis  of  Variance  For  Benchmarking  Process  Costs 


Job  Classification 

DF 

2d2 

a2 

F 

P 

Between  Groups 

8 

11.91 

1.49 

2.09 

.04 

Within  Groups 

120 

85.30 

.71 

DoD  Component 

DF 

£d2 

O2 

F 

P 

Between  Groups 

6 

10.55 

1.76 

2.95 

.01 

Within  Groups 

111 

66.07 

.60 

Source:  Developed  by  researcher. 

Regression  analysis  for  this  question  displayed 
relationships  between  the  respondent's  level  of  mathematics 
education  and  their  answers.  The  mathematical  nature  of 
parametric  cost  estimating  methods  led  to  the  belief  that  as 
one's  mathematics  education  increases  so  does  familiarity  with 
the  concepts  employed  in  them.  This  in  turn  should  lead  to 
greater  confidence  in  the  proper  application  of  the  methods  by 
these  same  individuals.  There  also  was  a  relationship  between 
General  Service/Senior  Executive  Service  (GS/SES)  level  or 
military  rank  for  Federal  Government  employees  and  their 
responses.  In  both  of  these  regression  analyses,  the  groups 
observed  agreed  with  the  main  body  of  respondents  in  the 
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degree  of  utility  for  parametric  cost  estimating  methods  in 
benchmarking  process  costs.  These  two  groups  displayed  a 
specific  trend  beyond  that  initial  observation,  however. 

The  regression  analysis  for  the  mathematics  education 
relationship  yielded  a  p  value  of  .02  for  the  six  subgroups 
reviewed.  This  value  indicates  a  relationship  does  exist 
between  the  data  points,  which  can  be  described  as  weak  in  the 
following  equation: 

Y  =  -.337(X)  +  38.3         R2  =  .033 

(.14)      (.40) 

(The  values  in  parentheses  represent  standard  errors  of  the 
above  coefficients.) 

In  this  analysis,  as  an  individual's  mathematics  education 

increases  the  numerical  value  of  their  responses  decreases. 

A  lower  value  indicates  greater  utility  because  of  the  inverse 

construction  of  the  response  scale  used  where  a  one  indicated 

the  greatest  utility  and  a  five  the  least  utility.   The  small 

value  of  the  coefficient  of  determination,  R2  ,  shows  that 

little  of  the  variation  in  the  dependent  variable,  responses, 

is  due  to  the  independent  variable,  mathematics  education. 

Similar  results  were  observed  in  the  analysis  comparing 

the  responses  with  the  GS/SES  level  or  military  rank.   The 

regression  equation  for  this  analysis 

Y  =  -.445(X)  +  8.68         R2  =  .134 

(.21)     (2.98) 
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shows  that  as  seniority  increases,  so  does  the  respondent's 
opinion  of  the  utility  of  parametric  cost  estimating  for 
benchmarking  process  costs.  This  relationship  can  also  be 
described  as  weak. 

Although  there  were  a  large  number  of  people,  who  did  not 
choose  to  offer  an  opinion,  most  of  those  that  did  saw  some 
promise  in  using  parametric  cost  estimating  methods  for 
benchmarking  process  costs.  Groups  indicating  a  high  degree 
of  utility  for  them  also  are  typically  in  an  oversight,  upper 
management  position  or  those  working  to  directly  support 
others  in  those  positions.  These  individuals  are  more  likely 
to  view  a  problem  in  a  systems  or  process  approach  and  may  be 
more  experienced  with  using  parametric  cost  estimating  methods 
in  the  area  of  benchmarking  process  costs  than  the  group  as  a 
whole. 

This  area  of  cost  analysis  is  one  that  may  take  on  more 
prominence  in  the  future  with  parametric  cost  estimating 
methods  playing  a  significant  role.  As  DoD  reliance  on 
military  specifications  and  standards  is  reduced,  the  cost 
estimating  techniques  built  on  those  standards  will  become 
less  reliable.  Greater  dependence  on  performance 
specifications  may  require  the  DoD  acquisition  community  to 
analyze  commercial  processes  that  have  had  little  or  no  prior 
application  in  DoD  acquisition  processes.    The  DoD  Single 
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Process  Initiative  (SPI)  may  provide  opportunities  to  use 
parametric  cost  estimating  methods  to  benchmark  process  costs. 
With  supervisory  personnel  experienced  in  and  amenable  to 
their  use,  parametric  cost  estimating  methods  should  see 
additional  future  applications  in  benchmarking  process  costs. 

2.    Validating  Bottoms  Up  Estimates 

Traditionally,  using  parametric  cost  estimating  methods 
in  this  area  involves  comparisons  between  the  parametric  cost 
estimate  and  the  project's  bottoms  up  estimate.  With  a  long 
history  of  use  in  these  applications,  the  aggregate  data 
showed  a  strong  trend  toward  acceptance  of  the  idea  that 
parametric  cost  estimating  methods  were  somewhat  useful  to 
validate  bottoms  up  estimates.  Only  20  responses  were  coded 
for  no  use  or  a  hindrance  to  validating  bottoms  up  estimates 
while  there  were  140  responses  indicating  opinions  that  the 
methods  were  at  least  somewhat  useful  in  these  situations. 
Group  breakdown  data  of  interest  are  presented  in  Table  5 
while  Table  6  contains  analysis  of  variance  data. 

Examination  of  the  analysis  of  variance  data  showed  those 
identifying  themselves  as  engineers  and  program  managers 
believed  that  parametric  cost  estimating  methods  were  more 
useful  than  did  the  other  job  classification  groups.  There  is 
a  significant  difference  between  the  upper  level  management's 
opinion  of  their  utility  and  that  of  their  supporting 
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organizations.   Support  groups  are  represented  by  the  program 
staffer  and  engineer  group  data  in  Table  5. 


Table  5 
Validating  Bottoms  Up  Estimates  Group  Data 


Min 

Max 

F 

Median 

a 

Ql 

Q3 

Program  Staff 

3.00 

3.00 

3.00 

3.00 

.00 

3.00 

3.00 

Corporate 
Policy 

1.00 

2.00 

1.40 

1.00 

.55 

1.00 

2.00 

Engineers 

1.00 

4.00 

2.88 

3.00 

1.1 

2.00 

4.00 

Program 
Managers 

1.00 

3.00 

1.50 

1.00 

.84 

1.00 

2.25 

Source:  Developed  by  researcher. 


Table  6 
Analysis  of  Variance  For  Validating  Bottoms  Up  Estimates 


Job  Classification 

DF 

2d2 

a2 

F 

P 

Between  Groups 

9 

15.37 

1.71 

2.45 

.01 

Within  Groups 

144 

100.53 

.70 

Source:  Developed  by  researcher. 

The  program  managers  and  those  who  set  corporate  policy 
felt  that  the  methods  were  vital  or  very  useful  in  this  area 
while  the  engineers  and  program  staffers  thought  that  the 
methods  were  only  somewhat  useful  in  this  functional  area. 
This  divergence  of  opinion  may  be  a  result  of  program  staffers 
and  engineers  who  had  closer  involvement  with  the  techniques 
and  methods  than  the  program  managers  and  those  who  set 
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corporate  policy.  The  weak  negative  relationship  exhibited  in 

the  regression  analysis 

Y  =  -.119(X)  +  7.11         R2  =  .033 
(.05)     (1.90) 

supports  the  contention  that  the  upper  level  supervisors  have 

a  higher  opinion  of  the  method' s  utility  than  do  the  support 

organizations.     The   small   R2   value   indicates   a   weak 

relationship,  however. 

Parametric  cost  estimating  methods  will  continue  to  be 
used  to  validate  bottoms  up  estimates.  This  function  is  one 
of  the  traditional  uses  for  parametric  cost  estimating  for 
which  there  is  still  a  need.  A  large  part  of  the  DoD 
acquisition  community  is  familiar  with  their  application  in 
validating  estimates  derived  from  other  sources.  Certain 
situations  lend  themselves  to  the  development  of  bottoms  up 
estimates,  and  the  subsequent  need  to  have  them  validated 
against  some  standard  will  always  remain.  These  factors 
should  continue  to  encourage  the  use  of  parametric  cost 
estimating  methods  in  validating  bottoms  up  estimates. 

3.    Program  Estimates 

Developing  program  estimates  is  also  an  area  of 
application  for  parametric  cost  estimating.  Consideration  of 
the  total  costs  for  a  program  or  system  has  always  been  a 
concern  and  is  becoming  more  important  in  acquisition 
planning.    In  the  aggregate  analysis,  the  majority  of  the 
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participants  felt  that  the  methods  were  useful  to  some  degree 
in  developing  program  costs.  Only  seven  respondents  answered 
that  there  was  no  use  for  parametric  cost  estimating  in 
program  estimating  and  no  one  responded  that  they  were  a 
hindrance.  The  average  response  value  for  the  entire  database 
was  2.25  with  a  standard  deviation  of  .82.  Demographic  group 
statistics  of  interest  are  shown  in  Table  7,  and  Table  8 
contains  analysis  of  variance  information. 

In  the  examination  of  the  analysis  of  variance  data  for 
job  classifications,  program  managers,  engineers,  and  contract 
negotiators  had  somewhat  lower  group  response  averages  than 
did  the  remaining  job  classification  groups  represented  in 
Table  7.  Analysis  of  variance  of  the  DoD  component  responses 
also  showed  some  interesting  trends.  The  DCAA  group's 
response  average  was  higher  than  that  of  every  other  group 
while  the  industry  group  had  the  lowest  response  averages. 
The  DCAA.  group  opinions  are  noteworthy  as  they  represent  a 
fairly  homogenous  response  from  a  large  subset,  33  responses 
to  this  question,  of  all  of  the  responses.  DCAA  has  a 
significant  part  in  the  DoD  acquisition  process.  They  have 
published  guidance  on  the  auditing  procedures  for  parametric 
cost  estimating  methods. 

The  NASA  responses  may  be  an  outgrowth  of  the  nature  of 
the  work  NASA  performs.   NASA  has  a  great  deal  of  experience 
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with  using  parametric  cost  estimates  in  numerous  developmental 
projects  and  may  be  more  experienced  in  their  use  than  the  DoD 
organizations.  The  NASA  respondents  also  have  one  of  the 
highest  average  levels  of  mathematics  education  of  the  groups 
surveyed.  The  DCAA  responses  may  reflect  distance  from  the 
program  estimating  functions  and  a  different  definition  of 
what  constitutes  program  estimating  than  that  used  in  the 
other  groups.  Also  of  note  is  that  the  DCAA  group  had  one  of 
the  lowest  mathematics  education  levels  of  the  groups 
surveyed.  These  factors  may  combine  to  produce  the  divergence 
of  opinion  in  the  utility  of  parametric  cost  estimating 
methods  between  these  two  groups. 

Table  7 
Program  Estimates  Group  Data 


Min 

Max 

M 

Median 

a 

Ql 

Q3 

Program 
Managers 

1.00 

3.00 

1.29 

1.00 

.76 

1.00 

1.00 

Industry 

1.00 

3.00 

1.47 

1.00 

.72 

1.00 

2.00 

Contract 
Negotiators 

1.00 

2.00 

1.50 

1.50 

.58 

1.00 

2.00 

NASA 

1.00 

4.00 

1.57 

1.00 

1.13 

1.00 

2.00 

Engineers 

1.00 

3.00 

1.78 

2.00 

.83 

1.00 

2.50 

DCAA 

1.00 

4.00 

2.76 

3.00 

.71 

2.00 

3.00 

Program 

Staff 

3.00 

3.00 

3.00 

3.00 

.00 

3.00 

3.00 

Source:  Developed  by  researcher. 
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Table  8 
Analysis  of  Variance  For  Program  Estimates 


Job  Classification 

DF 

2d2 

a2 

F 

P 

Between  Groups 

8 

19.17 

2.40 

4.43 

.00 

Within  Groups 

141 

76.33 

.54 

DoD  Component 

DF 

sd2 

a2 

F 

P 

Between  Groups 

6 

14.11 

2.35 

4.41 

.00 

Within  Groups 

132 

70.32 

.53 

Source:  Developed  by  researcher. 

The  data  and  statistics  for  this  question  appear  to  have 
contradictory  indications.  The  program  manager  and  program 
staff  groups  had  different  opinions  of  the  utility  of 
parametric  cost  estimating  methods;  the  program  managers  felt 
they  were  vital  or  very  useful  while  the  program  staff  felt 
that  they  were  only  somewhat  useful.  The  experience  and 
mathematics  education  response  values  for  these  two  groups  may 
explain  this  divergence.  The  program  managers  have  a 
significantly  higher  mathematics  education  level  on  average 
than  do  the  program  staff  but  a  lower  experience  average. 

The  relationships  between  the  two  regression  analyses 
seems  to  be  contradictory.  Two  of  the  groups  with  the  most 
experience  have  widely  divergent  opinions  about  the  utility  of 
the  methods  in  program  estimating.  One  would  expect  that  as 
experience  increases  so  would  someone's  familiarity  with 
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parametric  cost  estimating  methods,  their  uses  and 
limitations.   The  experience  regression  equation 

Y  =  .261 (X)  -  5.13         R2  =  .053 

(.08)     (2.39) 

indicates  a  weak  positive  relationship  exists.  More 
experience  leads  to  a  belief  that  the  methods  are  less  useful 
in  program  estimating.  The  mathematics  education  regression 
analysis 

Y  =  -.118 (X)  -  7.03         R2  =  .033 

(.05)     (1.75) 

shows  a  weak  negative  relationship.  The  DCAA  responses 
exemplify  these  two  equations.  Their  view  of  the  utility  of 
the  methods  in  this  functional  area  is  low  when  compared  to 
the  other  groups,  and  they  are  one  of  the  most  experienced 
groups,  but  they  also  have  one  of  lower  mathematics  education 
averages  of  the  groups. 

The  DCAA  group  had  lower  opinions  of  the  utility  of 
parametric  cost  estimating  methods  in  this  application  than 
did  the  other  groups.  This  may  represent  a  more  detailed 
working  knowledge  of  the  uses  of  the  methods  and  their 
limitations  than  the  other  groups  had.  It  also  may  be  a 
result  of  a  different  view  of  what  is  included  in  the 
definition  of  a  program  estimate.  These  groups  may  have  used 
a  more  restrictive  definition  of  a  program  estimate  than  did 
the  remaining  groups.    In  restricting  the  definition  to 
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specific  functional  estimates  produced  at  various  times  during 
a  program  cycle,  they  may  have  been  more  skeptical  about  using 
parametric  cost  estimating  methods  than  the  program  managers 
who  may  have  taken  a  more  expansive  view  of  the  definition. 
The  majority  of  the  groups  in  Table  7  show  a  trend  towards 
acceptance  of  the  use  of  these  methods  for  producing  program 
estimates. 

4 .    Budgeting 

Budgeting  is  another  area  in  which  parametric  cost 
estimates  have  a  long  history  of  use.  They  have  been  produced 
for  rough  order  of  magnitude  budgeting  as  well  as  for  more 
specific  estimates  in  the  budgeting  process. -  The  overall 
database  information  had  all  but  nine  participants  indicating 
their  belief  that  parametric  cost  estimating  had  some  use  in 
budgeting  decisions.  The  majority  of  the  respondents  chose 
the  descriptor  very  useful  as  their  response  to  the  question. 
The  average  response  value  was  2.2  with  a  standard  deviation 
of  .85  for  the  responses.  Table  9  displays  the  group 
breakdown  data  applicable  to  this  question  with  Table  10 
presenting  analysis  of  variance  data. 

The  subgroups  identified  as  cost  or  price  analysts  and 
contracting  officers  were  of  note  in  the  analysis  of  variance. 
Both  of  these  groups  represent  large  sections  of  the  database, 
45  and  36  responses  to  this  question  respectively,  with 
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relatively  small  standard  deviations  for  both  groups.   The 

U.S.  Army  and  DCAA  subgroups  were  also  identified  in  the 

analysis  of  variance  as  being  of  note.   These  two  groups  also 

represent  major  parts  of  the  database,  and  when  compared  with 

the  other  DoD  component  groups,  they  show  relatively  tight 

data  groupings  with  response  average  values  larger  than  those 

of  the  other  DoD  component  groups.   These  four  groups  all 

responded  that  the  use  of  parametric  cost  estimating  methods 

were  of  some  use  in  the  budgeting  process. 

Three   other   groups,   program   managers,    contract 

negotiators,  and  those  that  set  corporate  policy,  responded 

that  the  methods  are  vital  or  very  useful  to  the  budgeting 

process.   The  responses  from  the  program  managers  and  industry 

are  of  special  interest.   Of  the  groups  who  would  be  expected 

to  have  considerable  experience  in  budgeting,  they  have  the 

highest  opinion  of  the  utility  of  the  methods  in  budgeting. 

The  data  from  the  DCAA  and  program  manager  groups  are 

supported  by  the  mathematics  education  regression  analysis. 

The  equation 

Y  =  -.137  (X*)  +  7.33         R2  -  .051 
(.04)     (1.66) 

indicates  that  there  is  a  weak  negative  relationship 
demonstrated  in  the  DCAA  high  average  response  to  this 
question  and  relatively  low  mathematics  education  average  when 
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compared  with  other  groups .  The  program  managers  have 
opposite  characteristics  with  one  of  the  highest  mathematics 
education  levels  and  low  average  response  to  this  question. 


Table  9 
Budgeting  Group  Data 


Min 

Max 

M 

Median 

a 

Ql 

Q3 

U.S.  Army 

1.00 

4.00 

2.51 

3.00 

.73 

2.00 

3.00 

DCAA 

1.00 

4.00 

2.63 

3.00 

.77 

2.00 

3.00 

Cost  or 

Price 

Analysts 

1.00 

4.00 

2.23 

2.00 

.70 

2.00 

3.00 

Contracting 
Officers 

1.00 

4.00 

2.28 

2.00 

.78 

2.00 

3.00 

Program 
Managers 

1.00 

3.00 

1.57 

1.00 

.79 

1..00 

2.00 

Contract 
Negotiators 

1.00 

2.00 

1.25 

1.00 

.50 

1.00 

1.75 

Corporate 
Policy 

1.00 

2.00 

1.60 

2.00 

.55 

1.00 

2.00 

Source:  Developed  by  researcher. 

Reviewing  the  results  of  the  analysis  leads  one  to 
believe  that  parametric  cost  estimating  applications  in 
budgeting  should  continue  their  long  history  of  use.  As  the 
data  show,  there  are  a  large  number  of  groups  in  the  DoD 
acquisition  community  who  feel  strongly  that  parametric  cost 
estimating  methods  are  useful  techniques  for  budgeting.  The 
increased  prominence  that  they  are  given  through  the 
Parametric  Cost  Estimating  Initiative  will  widen  the  knowledge 
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about  their  uses  in  the  community.   The  increasing  accuracy 

and  speed  at  which  these  type  of  estimates  can  be  generated 

make  them  a  formidable  tool  for  contingency  analyses  in  an 

environment  of  funding  uncertainty.   From  their  already  wide 

acceptance  base,  this  should  lead  to  more  applications  in 

budgeting  in  the  future. 

Table  10 
Analysis  of  Variance  For  Budgeting  Data 


Job  Classification 

DF 

Ed2 

a2 

F 

P 

Between  Groups 

9 

20.97 

2.33 

3.97 

.00 

Within  Groups 

145 

85.22 

.59 

DoD  Component 

DF 

Sd2 

a2 

F 

P 

Between  Groups 

6 

10.48 

1.75 

2.92 

.01 

Within  Groups 

131 

78.34 

.60 

Source:  Developed  by  researcher. 

5.    Independent  Cost  Estimate  (ICE)  Generation 

Generating  independent  cost  estimates  is  an  important 
part  of  any  acquisition  planning  process.  They  may  be  used  in 
any  number  of  program  functions  to  include  program  estimates 
or  budgeting.  Parametric  cost  estimating  is  one  of  the  tools 
that  can  be  applied  to  generate  these  estimates.  Responses  to 
this  segment  were  centered  between  very  useful  and  some  use. 
No  responses  reflected  the  opinion  that  the  methods  would  be 
a  hindrance  to  generating  ICE  generation.  There  were  a  large 
number  of  no  responses  to  this  question,  however.  The  average 
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response  value  was  2.34  and  the  median  value  was  2,  or  very 
useful.  In  the  analysis  of  variance  the  engineers  believed 
that  the  use  of  parametric  cost  estimating  had  the  most 
application  to  ICE  generation  of  the  groups.  They  indicated 
that  the  methods  were  very  useful  while  the  remaining  job 
classification  groups  felt  that  the  methods  were  only  somewhat 
useful  for  ICE  generation.  Tables  11  and  12  contain  the 
group  specific  data  and  analysis  of  variance  for  this  segment 
of  question  one. 


Table  11 
ICE  Group  Data 


Min 

Max 

M 

Median 

a 

Ql 

Q3 

Control  Group 

1.00 

3.00 

2.00 

2.00 

.94 

1.00 

3.00 

USAF 

1.00 

3.00 

2.07 

2.00 

.73 

1.75 

3.00 

Engineers 

1.00 

3.00 

1.46 

1.00 

.69 

1.00 

2.00 

Source:  Develope 

:d  by  r 

esearc 

tier. 

Table  12 
Analysis  of  Variance  For  ICE  Data 


Job  Classification 

DF 

zd2 

a2 

F 

P 

Between  Groups 

9 

16.83 

1.87 

2.29 

.02 

Within  Groups 

104 

84.79 

.82 

Source:  Developed  by  researcher. 

The  responses  of  the  engineer  job  classification  group 
indicated  the  highest  level  of  regard  for  the  utility  of 
parametric  cost  estimating  for  producing  ICEs.   This  group  is 
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representative  of  those  in  the  DoD  acquisition  community 

tasked  with  ICE  generation  and  who  are  familiar  with  the 

process.     They  also   have   one   of   the   highest   average 

mathematics  education  levels  of  any  group  in  the  database. 

These   factors   indicate   a  viable   application  for  using 

parametric  cost  estimating  methods  in  ICE  generation  exists, 

and  there  is  a  base  of  expertise  in  their  use  in  this  area. 

The  high  regard  that  the  control  and  USAF  groups  had  for 

the  use  of  parametric  cost  estimating  methods   for  ICE 

generation  is  also  indicative  of  their  application  potential. 

The  control  group  has  been  studying  the  use  of  the  methods  and 

they  have  responded  that  the  techniques  are  very  useful  in 

this   area.    Regression  analysis  revealed  the  following 

relationship  between  answers  for  this  question  and  the 

respondents'  mathematics  education  levels. 

Y  =  -.169(X)  +  8.74         R2  =  .062 
(.06)     (2.14) 

The  weak  negative  relationship  between  the  answers  and  the 

mathematics  education  responses  supports  the  observation  that 

these  groups  would  strongly  support  the  use  of  parametric  cost 

estimating  methods  for  ICE  generation. 

The  application  of  parametric  cost  estimating  methods  to 

producing  ICEs  will  continue  and  become  more  prevalent.   DoD 

Regulation  5000. 2-R  states,  "As  defense  resources  decline,  the 

margin  available  for  adjustments  narrows,  making  realistic 
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cost  projections  all  the  more  critical."  [Ref  44]  Parametric 
cost  estimating  has  and  will  continue  to  fulfill  a  major  role 
in  producing  ICEs  to  meet  this  need.  The  groups  cited  in 
Table  11  are  important  members  of  the  DoD  acquisition 
community  who  favorably  view  the  application  of  parametric 
cost  estimating  methods  to  ICE  generation.  With  the  emphasis 
from  the  control  group  on  the  potential  uses  of  the  methods, 
the  engineers  representing  the  users  of  the  methods  in  this 
area,  and  the  auditor  group  noting  that  the  methods  have  some 
application  in  the  production  of  ICEs,  they  should  certainly 
find  wider  use  in  future  procurement  actions  and  planning. 

6.    Software  Contracts 

There  are  a  host  of  databases  and  computer  programs  that 
were  developed  for  the  purpose  of  estimating  costs  and 
schedules  for  software  development.  Examples  such  as  COCOMO 
and  PRICE-S  were  cited  earlier  in  the  text.  The  cumulative 
data  showed  that  the  largest  part  of  the  respondents  indicated 
that  parametric  cost  estimating  methods  had  some  use  in 
software  contracts.  Few  of  the  participants  (27)  thought  that 
they  had  no  use.  The  average  response  value  to  this  question 
was  2.54  and  a  median  value  of  3  was  observed  from  the 
aggregate  data. 

In  reviewing  the  group  data,  the  USAF  was  observed  to 
have  a    somewhat  higher  regard  for  the  application  of 
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parametric  cost  estimating  methods  to  software  contracting 
than  the  main  body  of  the  participants.  USAF  group  data,  an 
average  response  value  of  2.07  with  a  .616  standard  deviation 
and  median  value  of  2,  indicate  a  view  that  the  methods  are 
very  useful  in  this  area.  The  USAF  has  been  greatly  involved 
in  the  use  of  parametric  cost  estimates  for  estimating 
software  development  costs.  The  USAF  Cost  Analysis  Agency  has 
produced  a  revised  COCOMO  model  for  this  express  purpose. 
This  versatile  model  can  be  used  to  produce  a  variety  of 
outputs  throughout  the  life  cycle  of  a  software  product. 

Parametric  cost  estimating  has  a  strong  base  of 
application  as  a  tool  in  the  area  of  software  cost  estimating. 
Estimating  the  costs  of  and  schedule  for  software  development 
has  been  a  historically  difficult  area.  The  increasing 
importance  of  software  in  DoD  systems  and  equipment  could  make 
this  a  growth  area  for  the  future.  Parametric  cost  estimating 
methods  should  become  more  useful  for  software  cost  estimate 
generation  as  databases  become  more  interconnected. 

7 .    Repair  Parts  Contracts 

Aggregate  data  analysis  presented  a  view  that  parametric 
cost  estimating  methods  had  some  utility  in  producing  cost 
estimates  for  repair  parts  contracts.  The  average  response 
value  was  2.85,  a  median  value  of  3  and  a  normal  distribution 
of  the  data  around  that  value  was  observed.   Table  13  shows 
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the  group  data  for  contracting  officers  and  those  that  set 
Government  policy.  The  groups  shown  are  noted  for  their 
concurrence  within  their  groups  demonstrated  by  the  small 
standard  deviations  identified  for  each  group.  Both  groups 
can  also  potentially  play  a  major  part  in  .the  use  of 
parametric  cost  estimating  in  repair  parts  contracting.  Those 
that  set  Government  policy  (through  regulation  or  directive) 
and  contracting  officers  (through  applying  the  methods  in 
specific  procurement  actions)  have  the  potential  to  directly 
encourage  or  discourage  the  use  of  parametric  cost  estimating 
methods  in  this  area.  That  these  groups  both  indicate  that 
they  see  limited  use  for  the  methods  in  repair  parts 
contracting  would  seem  to  limit  their  uses  here. 

Repair  parts  contracting  is  not  normally  an  area  in  which 
the  methods  of  parametric  cost  estimating  are  applied. 
Generally  the  costs  or  prices  are  fairly  well  established 
through  historical  prices.  The  need  for  estimating  repair 
parts  cost  or  prices  typically  occurs  later  in  the  life  cycle 
than  the  stages  in  which  parametric  cost  estimating  methods 
are  used.  Parametric  cost  estimating  methods  could  be  used  in 
an  overall  estimate  of  repair  parts  costs  for  a  system  or 
piece  of  equipment  much  more  readily  than  for  an  individual 
procurement  action.   The  data  reflect  a  preference  for  other 
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cost  or  price  estimating  methods  to  estimate  repair  parts 
costs  over  the  use  of  parametric  cost  estimating  methods. 


Table  13 
Repair  Parts  Contracting  Group  Data 


Min 

Max 

M 

Median 

o 

Ql 

Q3 

Contracting 
Officers 

2.00 

4.00 

2.85 

3.00 

.57 

2.50 

3.00 

Government 
Policy 

2.00 

4.00 

3.05 

3.00 

.67 

3.00 

3.50 

Source:  Develope 

id  by  r 

esearc 

her. 

8.    R&D  Contracts 

R&D  contracting  is  an  area  in  which  parametric  cost 
estimating  can  be  applied  in  a  number  of  areas.  The  types  of 
items  and  services  procured  in  R&D  contracting  are 
developmental  in  nature,  and  estimating  costs  or  prices  can  be 
difficult.  The  database  analysis  for  the  entire  response 
group  yielded  an  almost  equal  distribution  between  the  very 
useful,  somewhat  useful  and  no  use  descriptors,  41,  49,  and  40 
responses  respectively.  An  average  response  value  of  2.83 
indicates  a  somewhat  useful  viewpoint  throughout  the  DoD 
acquisition  community  regarding  the  utility  of  parametric  cost 
estimating  methods  to  R&D  contracting.  The  NASA  participants 
had  the  highest  regard  for  their  applicability  in  the  analysis 
of  variance.   Job  classification  and  DoD  component  data  for 
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the  analysis  of  variance  is  displayed  in  Table  15.  DCAA 
responses  indicated  that  in  their  view  the  methods  were  of 
some  or  no  use  to  R&D  contracting. 


Table  14 
R&D  Contracting  Group  Data 


Min 

Max 

M 

Median 

a 

Ql 

Q3 

DCAA 

2.00 

5.00 

3.53 

4.00 

.80 

3.00 

4.00 

USAF 

1.00 

4.00 

2.21 

2.00 

.89 

2.00 

2.25 

NASA 

1.00 

3.00 

1.67 

1.50 

.82 

1.00 

2.25 

Technical 
Support 

3.00 

4.00 

4.00 

4.00 

1.00 

3.00 

4.00 

Source:  Developed  by  researcher. 

The  NASA  and  USAF  data  indicate  that  there  are  some  in 
the  DoD  acquisition  community  that  feel  that  parametric  cost 
estimating  methods  can  be  used  in  R&D  contracting.  It  is  also 
of  interest  that  the  technical  support  group  participants  were 
the  group  that  had  the  highest  average  response  rate  of  all 
the  groups.  The  response  rate  shown  in  Table  14  indicates 
that  this  small  group  believes  that  the  methods  have  no  use  in 
R&D  contracting. 

The  mathematics  education  regression  analysis  supports 
the  observations  regarding  the  groups  in  Table  14.  The 
regression  equation 


Y  =  -.200(X)  +  10.3 
(.05)     (2.01) 


R2  =  .07  8 
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indicates  that  there  is  a  weak  negative  relationship  between 
mathematics  education  and  the  responses  for  this  question. 
The  NASA  participants  had  a  higher  average  mathematics 
education  level  than  did  the  other  groups  and  they  also  felt 
that  the  methods  had  more  use  in  R&D  contracting  than  did  the 
other  respondents. 


Table  15 
Analysis  of  Variance  For  R&D  Contracting  Data 


Job  Classification 

DF 

Ed2 

G2 

F 

P 

Between  Groups 

9 

31.44 

3.49 

3.62 

.00 

Within  Groups 

138 

133.29 

.97 

DoD  Component 

DF 

Ed2 

a2 

F 

P 

Between  Groups 

6 

28.32 

4.72 

5.42 

.00 

Within  Groups 

127 

110.61 

.87 

Source:  Developed  by  researcher. 

The  application  of  parametric  cost  estimating  methods 
should  continue  to  be  used  in  this  area  where  the  groups 
mentioned  have  found  them  useful.  Increasing  budgetary 
pressures  have  restricted  spending  on  R&D  throughout  DoD  and 
it  is  likely  that  R&D  funding  will  be  restricted  in  the 
future.  A  lack  of  useful  databases  in  the  R&D  field  may  be 
contributing  to  the  view  of  almost  one  third  of  the 
respondents  that  parametric  cost  estimating  methods  are  of  no 
use.   While  parametric  cost  estimating  cannot  be  used  for 
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every  instance,  the  methods  do  have  the  potential  to  produce 
sound  estimates  for  R&D  applications. 

9.    Service  Contracts 

Acquisition  of  services  forms  a  significant  part  of  the 
Federal  Government's  procurement  activity.  Parametric  cost 
estimating  methods  have  not  been  widely  applied  to  this  area 
of  acquisition  in  the  past.  The  majority  of  the  responses  to 
this  segment  were  split  between  some  use  and  no  use  with  57 
responses  for  each  choice.  The  average  response  value  from 
the  database  was  3.09  and  the  median  response  value  was  3. 
These  indicate  that  the  participants  felt  there  was  only  some 
use  for  the  methods  in  service  contracting.  The  results  of 
the  analysis  are  shown  below  in  Table  16. 

Service  contracting  usually  does  not  have  the 
characteristics  that  make  it  amenable  to  using  parametric  cost 
estimating  techniques.  The  procurements  in  service 
contracting  are  not  often  quantifiable  in  terms  of  parameters 
that  are  easily  measured  and  then  related  to  outputs  in  a 
database.  There  are  few  existing  databases  and  software 
products  that  are  used  in  this  area.  The  control  group 
average  response  rate  supports  this  view  in  their  opinion  that 
parametric  cost  estimating  methods  do  not  have  any  use  in  the 
area  of  service  contracting.  The  rather  large  standard 
deviation  indicates  a  lack  of  consensus  within  this  group, 
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however,  but  at  no  point  does  anyone  in  the  group  advocate 

that  the  methods  have  any  more  than  some  use  in  service 

contracting. 

Table  16 
Service  Contracting  Group  Data 


Min 

Max 

V 

Median 

a 

Ql 

Q3 

Set  Corporate 
Policy 

3.00 

4.00 

3.25 

3.00 

.50 

3.00 

3.75 

Control  Group 

3.00 

5.00 

4.17 

4.00 

.98 

3.00 

5.00 

Source:  Developed  by  researcher. 

The  group  of  respondents  that  set  corporate  policy 
differs  in  their  opinion  but  only  slightly.  Their  answers 
lead  one  to  believe  that  they  feel  that  the  methods  have  some 
use  in  this  area  of  contracting.  Unlike  the  control  group, 
those  that  set  corporate  policy  have  a  closer  consensus 
regarding  the  methods'  application  in  service  contracting 
evidenced  by  the  small  standard  deviation  in  a  group  twice  the 
size  of  the  control  group. 

Defining  services  to  be  acquired  by  performance  standards 
may  allow  greater  use  of  parametric  cost  estimating  methods  in 
service  contracting.  Performance  specifications  are  the 
preferred  method  of  describing  supplies  or  services  required 
by  the  Federal  Government.  With  this  emphasis  and  increased 
awareness  of  parametric  cost  estimating  methods,  the 
techniques  may  see  more  use  in  this  area  of  Federal  Government 
procurement . 

93 


10.   Hardware  Contracts 

Although  hardware  can  be  defined  in  terms  of  components, 
assemblies  or  subassemblies  of  an  electrical,  mechanical,  or 
electronic  piece  of  equipment,  the  definition  used  here  is 
more  limited.  Hardware  contracting  for  this  study  refers  to 
contracting  for  computer  hardware,  equipment,  or  peripherals. 
The  rapid  pace  of  development  in  the  computing  field  leads  to 
the  use  of  parametric  cost  estimating  in  this  area.  The  data 
and  analysis  apply  equally  as  well  to  a  broad  definition  of 
hardware  as  they  do  to  the  narrower  definition  used  in  this 
study. 

Database  analysis  revealed  that  the  majority  of  the 
respondents  answered  that  the  methods  were  very  useful  or  of 
some  use  in  hardware  contracting,  54  and  59  responses 
respectively.  Some  use  was  the  median  response  and  an  average 
response  value  was  2.55.  Contracting  officers  were  the  only 
group  that  showed  a  consensus,  a  standard  deviation  of  .62  was 
observed,  in  their  responses  to  this  question.  In  their 
position,  they  are  also  one  of  the  groups  that  could  best 
apply  the  methods  to  contracting  for  hardware.  Their 
responses  with  a  mean  of  2.65  and  median  value  of  3  indicate 
a  view  that  the  methods  are  of  some  use  in  this  area. 

While  the  methods  may  have  application  in  producing 
estimates  to  be  used  in  the  area  of  hardware  contracting,  the 
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drive  to  use  Commercial-of f-the-Shelf  Technology  (COTS)  and 
Non-developmental  Items  (NDI)  may  restrict  their  uses  by  the 
Federal  Government  in  this  area  in  the  future.  COTS  items  and 
NDI  are  by  definition  beyond  the  stage  in  which  parametric 
cost  estimating  methods  are  typically  applied  in  the  product 
life  cycle,  as  shown  in  Figure  1.  If  the  item  is  truly  a  COTS 
item  or  NDI  there  should  be  ample  pricing  data  available  from 
the  commercial  environment  and  less  need  to  apply  the 
techniques  to  this  area  of  Federal  Government  acquisition. 
This  type  of  acquisition  environment  may,  however,  hold  some 
promise  for  the  use  of  parametric  cost  estimating  methods. 
The  methods  may  be  used  to  estimate  the  costs  of  integrating 
the  various  COTS  items  and  NDI  that  may  be  purchased  to  meet 
an  agency's  needs  rather  than  for  the  equipments  themselves. 

11.   Construction  Contracts 

Parametric  cost  estimating  methods  have  been  successfully 
applied  to  many  construction  projects.  There  are  plentiful 
data  and  output  parameters  in  construction  that  lend 
themselves  to  using  parametric  cost  estimating  methods.  The 
CPRG  uses  a  construction  example  to  illustrate  the  development 
of  concepts  and  CERs  in  parametric  cost  estimating.  Some  use 
was  the  choice  of  54  of  the  respondents  to  the  survey.  There 
were  a  significant  number  of  participants  (79)  who  elected  to 
not  respond  to  this  segment  of  question  one.   An  average 
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response  value  for  the  participants  of  2.71  was  observed  and 
the  median  response  was  the  some  use  descriptor.  The  group 
response  data  are  presented  below  in  Table  17. 


Table  17 
Construction  Contracting  Group  Data 


Min 

Max 

M 

Median 

o 

Ql 

Q3 

Contracting 
Officers 

2.00 

4.00 

2.79 

3.00 

.78 

2.00 

3.00 

Negotiators 

1.00 

2.00 

1.75 

2.00 

.50 

1.25 

2.00 

Government 
Policy 

2.00 

4.00 

2.86 

3.00 

.72 

2.00 

3.00 

Source:  Developed  by  researcher. 

The  contract  negotiator  response  is  of  particular 
interest  in  Table  17.  This  group  indicated  that  they  believed 
that  there  was  greater  utility  for  parametric  cost  estimating 
methods  in  construction  contracting  than  did  the  other 
participants  and  groups.  The  contract  negotiators  indicated 
the  methods  were  vital  or  very  useful  in  the  area  of 
construction  contracting  while  others  felt  that  they  were  only 
of  some  use  in  the  process. 

The  contract  negotiators'  consensus  on  this  question  is 
also  noteworthy  when  compared  with  the  responses  from  the 
contracting  officers.  With  similar  training  and  experiences, 
one  would  expect  similar  responses  from  both  groups.  The 
contracting  officers,  however,  felt  that  the  methods  were  only 
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of  some  use  in  construction  contracting.  These  individuals 
and  those  that  set  Government  policy  can  be  most  influential 
in  applying  the  methods  in  this  area,  and  both  do  not  see  the 
methods  contributing  significantly  in  this  area. 

Parametric  cost  estimating  methods  should  continue  to  be 
employed  in  estimating  for  construction  contracting.  They 
have  an  established  basis  for  use  outside  the  DoD  and  a  large 
part  of  the  DoD  acquisition  community  involved  in  construction 
contracting  is  aware  of  how  they  can  be  best  applied  to  this 
area.  Base  consolidations  and  relocations  could  offer 
additional  opportunities  for  using  parametric  cost  estimating 
methods  in  construction  projects. 

12.   Commodity  Contracts 

Contracting  for  commodities,  such  as  fuels,  clothing,  or 
food  items,  is  a  significant  and  vital  part  of  the  DoD 
acquisition  workload.  Commodities  may  defined  as  commercial 
items  sold  in  substantial  quantities  to  the  general  public 
exemplified  by  the  group  and  class  categories  in  the  GSA 
catalogs.  Aggregate  group  analysis  showed  that  the  majority 
of  the  responses  to  this  question  were  split  between  some  use 
and  no  use  in  this  area,  51  and  41  responses  respectively. 
The  analysis  also  showed  a  median  response  value  of  3  and 
average  response  value  of  3.23  with  a  standard  deviation  of 
.78.   The  small  standard  deviation  indicates  that  a  consensus 
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on  this  question  exists.   No  group  in  the  analysis  deviated 
from  this  opinion. 

Pricing  for  commodities  is  normally  accomplished  using 
market  pricing  mechanisms.  As  a  result,  there  are  few 
opportunities  for  the  use  of  parametric  cost  estimates.  The 
aggregate  group  data  presented  support  the  idea  that  the 
methods  have  limited  application  in  commodity  contracting. 
This  situation  is  unlikely  to  change  significantly  because 
market  prices  are  one  of  the  preferred  pricing  methods  for 
commodity  items. 
C.    SUMMARY 

Question  one  was  broken  into  12  segments  to  -take  a  broad 
view  of  some  of  the  major  areas  in  the  acquisition  process  to 
which  parametric  cost  estimating  may  be  applied.  The  results 
of  the  analysis  conducted  across  these  areas  showed  the 
opinions  of  the  DoD  acquisition  community  regarding  the 
applicability  of  the  methods.  Overall,  the  methods  were  seen 
to  be  most  applicable  to  the  program  estimating,  budgeting  and 
ICE  generation  and  least  applicable  to  commodity  contracting. 
In  all  of  the  other  areas  the  respondents  indicated  there  was 
some  use  for  the  techniques. 
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VI.    SPECIFIC  APPLICATIONS 

A.  PURPOSE 

Questions  two  through  four  ask  the  participants  to  rank 
the  utility  of  parametric  cost  estimating  methods  across  a 
series  of  specific  applications  in  program  management  and 
contract  situations.  Question  two  addresses  system  life  cycle 
concerns  as  defined  in  DoD  Regulation  5000. 2-R.  Figure  1 
presents  a  typical  pattern  of  applying  the  methods  throughout 
the  four  life  cycle  phases.  Question  three  presents  data  on 
the  uses  of  parametric  cost  estimating  methods  in  specific 
contract  type  situations.  Finally,  question  four  addresses 
applications  in  pre-award  and  post-award  contract  actions. 

B.  PROGRAM  LIFE  CYCLES  PHASES 
1 .    Concept  Exploration 

In  the  first  phase  of  a  system's  life  cycle,  concept 
exploration  (CE) ,  a  series  of  short  term,  independent  studies 
are  conducted.  These  studies  attempt  to  evaluate  various 
concepts  that  are  proposed  to  meet  a  requirement.  Parametric 
cost  estimating,  as  shown  in  Figure  1,  is  not  the  primary  cost 
estimating  method  in  this  phase.  The  aggregate  responses  were 
essentially  evenly  spread  across  the  three  utility  descriptors 
and  no  responses.  Vital,  very  useful,  some  use,  and  no 
responses  garnered  39,  48,  43,  and  39  answers  respectively. 


99 


The  dispersion  of  responses  indicates  a  lack  of  agreement  on 
the  answer  to  this  question  which  is  also  identified  in  the 
large  standard  deviation  of  1.09.  The  average  response  value 
of  2.46,  a  median  value  of  2  and  a  positively  skewed 
distribution  indicate  that  the  participants  felt  that  the 
methods  were  very  useful  in  this  life  cycle  phase. 

The  job  classification  analysis  of  variance  data 
highlighted  the  control  and  NASA  groups.  Their  responses 
indicated  that  for  this  phase  they  felt  the  methods  had  the 
most  utility  of  all  of  the  groups.  The  auditor  job 
classification  group  was  identified  as  believing  that  the 
methods  had  the  least  degree  of  utility  in  this  life  cycle 
phase.  These  results  may  be  an  outcome  of  the  experience  that 
the  control  and  NASA  groups  have  with  this  phase  of  program 
management  and  the  lesser  degree  of  experience  that  the 
auditors  may  have  with  it.  Tables  18  and  19  contains  these 
data. 

The  low  response  averages  for  the  program  staff,  NASA, 

and  control  groups  for  this  question  are  noteworthy.   These 

groups  represent  high  mathematics  education  levels  and  are 

familiar  with  the  actions  required  in  each  of  the  phases  of  a 

system's  life  cycle.   The  mathematics  regression  analysis  also 

supports  this.   The  regression  equation  and  data 

Y  =  -.271(X)  +  12.6         R2  =  .132 
(.05)     (1.99) 
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show  a  weak  negative  relationship  between  the  responses  and 

the  mathematics  education  level  of  the  participants.   These 

groups  indicated  that  they  believed  that  the  methods  were  very 

useful  in  the  concept  exploration  phase.    The  data  from 

analysis  of  the  auditor  and  contracting  officer  responses  show 

that  they  feel  that  the  methods  are  only  of  some  use  in  this 

phase. 

Table  18 
Concept  Exploration  Group  Data 


Min 

Max 

M 

Median 

a 

Ql 

Q3 

Program 
Staff 

2.00 

2.00 

2.00 

2.00 

.00 

2.00 

2.00 

Control 
Group 

1.00 

3.00 

1.45 

1.00 

.69 

1.00 

2.00 

NASA 

1.00 

4.00 

1.50 

1.00 

1.07 

1.00 

1.75 

Auditors 

1.00 

4.00 

3.13 

3.00 

.91 

2.00 

4.00 

Source:  Developed  by  researcher. 


Table  19 
Analysis  of  Variance  For  Concept  Exploration  Data 


Job  Classification 

DF 

2d2 

a2 

F 

P 

Between  Groups 

9 

28.99 

3.22 

2.97 

.00 

Within  Groups 

149 

161.35 

1.08 

DoD  Component 

DF 

2d2 

a2 

F 

P 

Between  Groups 

6 

26.25 

4.37 

4.17 

.00 

Within  Groups 

134 

140.73 

1.05 

Source:  Developed  by  researcher 
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The  CE  phase  activity  is  centered  on  reviewing  various 
solutions  to  a  problem  vice  analyzing  a  specific  solution.  It 
is  not  until  a  general  course  of  action  is  decided  on  that 
system  measures  of  effectiveness  or  performance  can  be 
developed.  Without  some  general  definition  of  how  a  solution 
to  the  problem  is  to  be  described,  a  database  cannot  be 
selected  for  analysis  nor  are  there  any  parameters  to 
evaluate.  Once  this  has  been  accomplished,  parametric  cost 
estimating  methods  can  be  successfully  used  in  this  area  of 
program  management.  In  this  phase,  they  may  have  more 
application  as  rough  order  of  magnitude  planning  tools  in 
budgeting  or  for  conducting  cost  and  performance  tradeoff 
analyses  early  in  a  program  than  for  specific  cost  or  schedule 
estimates . 

2.  Program  Definition  &  Risk  Reduction 
Activity  in  the  PD&RR  phase  centers  on  the  evaluation  of 
the  problem  solution  identified  at  the  Milestone  I  decision 
point.  Defined  in  DoD  Regulation  5000. 2-R,  these  activities 
may  include  prototyping,  testing,  and  early  operational 
assessment  of  critical  systems,  subsystems,  and  components. 
[Ref  43:p.4]  Actions,  such  as  these,  are  taken  to  identify 
and  reduce  risk  in  the  early  stages  of  a  program.  Responses 
to  this  segment  of  question  two  were  concentrated  on  very 
useful,  60  responses,  and  some  use,  58  responses.   An  average 
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response  value  of  2.43  and  median  answer  of  very  useful  was 
observed. 

Analysis  of  variance  results  highlighted  the  corporate 
policy,  NASA,  program  staff,  and  control  groups  as 
particularly  interesting.  These  groups  believe  that  the 
methods  are  vital  to  this  phase  of  a  system's  life  cycle.  In 
fact,  the  program  staff  members  were  unanimous  in  their 
opinion  that  the  methods  were  very  useful  in  the  PD&RR  phase. 
Data  analysis  results  are  displayed  in  Tables  20  and  21.  The 
view  expressed  by  the  groups  that  the  methods  have  a  great 
deal  of  utility  in  this  phase  is  noteworthy.  Most  of  the 
groups  work  in  this  area  of  program  management  or  have 
experience  in  performing  the  functions  in  this  phase  and 
represent  a  cross  section  of  upper  level  management  in  the  DoD 
acquisition  community. 

Table  20 
PD&RR  Group  Data 


Min 

Max 

M 

Median 

a 

Ql 

Q3 

Program 
Staff 

2.00 

2.00 

2.00 

2.00 

.00 

2.00 

2.00 

Control 
Group 

1.00 

2.00 

1.55 

2.00 

.52 

1.00 

2.00 

Corporate 
Policy 

1.00 

2.00 

1.20 

1.00 

.45 

1.00 

1.50 

NASA 

1.00 

4.00 

1.57 

1.00 

1.13 

1.00 

2.00 

Source:  Developed  by  researcher. 
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Figure  1  presents  the  view  that  parametric  cost 
estimating  is  most  useful  in  this  stage  of  a  system's  life 
cycle.  This  is  demonstrated  by  the  response  averages  and 
standard  deviations  of  the  groups  shown  in  Table  20.  The 
following  regression  eguations  support  this  observation. 


Mathematics  Education 

Y  =  -.201 (X)  +  9.95 

(.05)     (1.70) 

GS/SES  Level/Rank 

Y  =  -.343(X)  +  7.09 

(.16)     (2.24) 


Rz   =  .099 


&   =  .094 


Although  these  relationships  can  be  described  as  weak,  they  do 
indicate  that  as  one's  mathematics  education  and  rank  increase 
the  belief  in  the  utility  of  parametric  cost  estimating 
methods  to  PD&RR  becomes  stronger. 


Table  21 
Analysis  of  Variance  For  PD&RR  Data 


Job  Classification 

DF 

Ed2 

O2 

F 

P 

Between  Groups 

9 

15.58 

1.73 

2.12 

.03 

Within  Groups 

153 

124.73 

.82 

DoD  Component 

DF 

Ed2 

O2 

F 

P 

Between  Groups 

6 

13.37 

2.23 

2.93 

.01 

Within  Groups 

138 

140.86 

.76 

Source:  Developed  by  researcher. 

Applications  for  parametric  cost  estimating  in  the  PD&RR 
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to  be  used  in  this  phase  and  become  more  important  as  they 
become  more  accurate  and  rapidly  generated.  There  is  a 
concurrence  in  the  opinions  regarding  their  utility  across  the 
DoD  acquisition  community  represented  by  the  corporate  policy. 
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professionals  should  ensure  that  they  are  used  where  they  will 
provide  the  most  benefit. 

3 .    Engineering  &  Manufacturing  Development 

in  me  iiingi.neerj.ng  &  rj.anuj.aci_urj.ng  j_>eve_.opmenu  pnase 

(E&MD)  ,  one  of  the  functions  is  to  define  and.  perfect  the 

manufacturing  processes  to  be  used  to  produce  the  equipment 

selected  at  the  Milestone  II  decision  point.   These  processes 

provide   opportunities   for   the   use   of   parametric   cost 
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estimates  also.  This  phase  is  the  transition  point  between 
the  defined  solution  and  production  of  equipment  representing 
that  solution. 

Participants  responded  to  this  question  with  62  responses 
in  the  very  useful  and  7  6  responses  in  the  some  use  descriptor 
categories.  There  was  an  average  response  value  of  2.45  and 
the  median  response  was  for  some  use  in  the  E&MD  phase. 
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DCAA  represented  this  overall  view  that  the  methods  had  only 
some  use  in  E&MD.  They  were  joined  in  this  belief  by  the  U.S. 
Army,  a  major  buying  organization  in  the  process. 

In  the  group  data,  the  responses  of  the  cost  estimators 
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corporate  policy  group  responses  show  a  greater  utility  for 
the  methods  here  than  do  the  cost  estimators.  The  cost 
estimators  believed,  as  did  the  DCAA  group,  that  the  methods 
have  limited  application  in  the  E&MD  phase  while  those  who  set 
corporate  policy  believe  that  it  is  vital  to  the  process  to 
use  them  in  the  E&MD  phase.  This  divergence  of  opinion 
between  these  groups  is  noteworthy  in  that  all  of  these  groups 
have  a  major  role  in  program  management  at  this  phase.  Tables 
22  and  23  contain  the  arouo  analvsis  and  analvsis  of  variance 


xaoie  // 
E&MD  Group  Data 


,                            ...      , , 1                     1 1 1 1 ■         1 

1  Min        1  Max      1  »i           1  Median    \  o        1  01         1  03 

j  r                                                      j   ~              j   ~ 

Cost                    '  1 .  00      1  3 .  QO    12.96    '3.00         '  .  66    '2.00    '3.00 

J— 1  O    i 1_  ILL  CX   L-  \~>  J-    o            j 

Corporate       '  1.00      1  2.00    '1.40    I  1.00         •  .55      1.00    '2.00 

X  uxx^j, 

DCAA                    ll.OO      1  4.00    12.79    13.00         1.70    12.00    13.00 

1                 1                1                1                    III                     1 

1                 III                    III 

U.S.    Army          1.00      |  4.00    j  2.61    J3.00         j  .70    |  2 . 00    j  3.00 
i i i i i i i i 1 
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applications  of  the  methods  in  the  same  functional  areas  by 
the  commercial  and  Federal  Government  sectors  of  the 
acquisition  community.  The  Federal  Government  interest  in  the 
application      of     business      practices      in      the      operation      of 
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usee,  witri  greater  frequency  in  tnis  pnase  ox  programs  oj.  an 
types. 

Tshlp  23 


Sr»o  1  uci   c       o  "P 

^7"^vt   or*k/*-»*__i       TT/^-k* 

TT  fMPi     Pio  -1-  = 

1 

I  DoD   Component 

i 

|  DF 
| 

Ed2 

1 

o2       If 

I              1 

IP 

|              j 

Between   Groups 

1  6 
1 

8.52 

1.42     12.51 

1  .03   I 

Within   Groups 

1 

|  139 

78.73 

1                II 

.5/ 

Source:  Developed  by  researcher. 

by  the  emphasis  on  cost- benefit  tradeoffs  that  are  now 
encouraged  by  DoD  upper  management  levels.   Evaluating  new 
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DoD  systems  also  provide  additional  avenues  for  using  the 
methods  more  widely  during  the  E&MD  phase.  Increasing 
accuracy  of  the  parametrically  derived  estimates  and 
increasina  costs  of  Droducina  bottoms  ur>  estimates  should  also 


107 


Support 

The  final  phase  described  in  DoD  Regulation  5000. 2-R  is 
the  Production,  Fielding/Deployment  &  Operational  Support 
phase  (PF/D&OS)  =   In  this  phase-  production  of  the  eguipment,- 

O^OUClll        »w»  J-         OUi-LWClLC         J-  O        -k-'^VJ  ULJ.1  /  ail'^l         LUC        v^  £-*  *c:  J-  a.  I Lunax         »-too        UJ.  L-J.XO 

product  starts.  Application  of  parametric  cost  estimates  in 
this  phase  have  been  restricted.  One  such  use  may  be  in 
verifying  that  the  program  is  tracking  with  earlier  cost 
estimates  by  comparing  them  with  actual  production  costs. 
Pes^onses  to  this  question  concentrate  on  vsrw  useful  -  52 

value  was  2 .  63  and  the  median  value  of  3  or  the  some  use 

descriptor  from  the  scale  used.   There  were  no  noteworthy 

group  data  for  this  guestion.   The  regression  analysis 

Y  =  .206(X)  -  3.20         R2  =  .024 
(.09)     (2.59) 

shows  a  positive  relationship  between  the  responses  and 

the  experience  level  of  the  participants-.   The  relationship 

between  experience  and  the  data  is  weak  although  it  does 

support   the   idea   that   the   experienced   groups   in   the 

acguisition  community  view  the  methods  as  having  limited  use 

in   the   PF/D&OS   phase.     Few   areas   of   application   can 

effectively  use  parametric  cost  estimating  methods  in  the 

PF/D&OS  ^hase.   Actual  costs  of  production  and  the  application 
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of  the  learning  curve  concept  over  the  number  of  units 
produced  are  the  major  cost  estimating  methods  used. 
Parametric  cost  estimating  methods  have  little  role  to  play 
and  are  unlikely  to  take  on  greater  prominence  in  this  area  in 
the  future. 

The  response  analysis  fits  the  pattern  of  use  for 
parametric  cost  estimating  shown  in  Figure  1.  The 
participants  as  a  group  believe  that  the  methods  have  some 
applications  in  the  CE  phase,  most  utility  in  the  PD&RR  and 
E&MD  phases  and  the  least  utility  in  the  PF/D&OS  phase.  The 
groups  with  the  most  experience  using  the  techniques  are  also 
the  strongest  proponents  of  their  utility  in  the  PD&RR  and 
E&MD  phases.  Future  improvements  to  the  techniques  that 
increase  their  accuracy  and  flexibility  should  combine  with 
the  discretion  given  to  decision  makers  to  make  sound  business 
decisions  to  increase  the  uses  of  parametric  cost  estimating 
techniques  in  all  of  the  program  life  cycle  phases. 
C.    CONTRACT  TYPE 

Ideally,  contract  type  is  directly  tied  to  the  risk 
inherent  in  a  particular  contract  situation.  These  situations 
are  in  part  formed  as  a  function  of  the  program  life  cycle 
stage  that  the  procurement  represents.  Figure  1  relates  the 
cost  risk  to  the  program  stages  over  time.  The  narrowing  cost 
uncertainty  band  can  be  interpreted  as  declining  cost  risk  as 
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one  progresses  through  the  life  cycle  of  a  program.  The 
contract  vehicles  typically  used  in  each  stage  also  progress 
from  ones  that  place  greater  risk  on  the  buyer  in  earlier 
phases  to  ones  that  shift  the  burden  of  risk  to  the  producer 
in  the  later  stages  of  the  program.  A  contract  type  pattern 
of  use  throughout  a  program  might  be  described  as  [Ref  41] : 

•  Phase  0    -  CPFF 

•  Phase  I    -  CPIF 

•  Phase  II   -  FPIF  or  CPAF 

•  Phase  III  -  FPIF  or  FFP 

One  of  the  goals  of  program  planning  is  to  match,  the  contract 
vehicle  to  the  risks  in  each  procurement  in  each  phase  of  the 
program.  Figure  8  describes  a  relationship  between  contract 
type,  risk,  and  the  utility  of  parametric  cost  estimating 
methods . 

Question  three  is  divided  into  segments  reflecting  each 
contract  type  depicted  in  Figure  8 .  The  response  data  for 
each  contract  type  segment  will  be  examined  to  determine  the 
relationship  between  this  theoretical  presentation  and  the 
reality  of  the  acquisition  environment.  Group  data  will  be 
presented  for  the  control,  program  manager,  industry,  and 
contracting  officer  groups  for  each  contract  type.  These 
groups  have  extensive  knowledge  about  and  experience  with 
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using   various   contract   types   in   different   procurement 
situations . 


Supplier 


Contract  Risk 


Contract  Risk 


Buyer 


Maximum 


PCE  Utility 


Minimum 


RFP  FPE  FPIS  FPR(P)  FPR(R)  FPIF  CPIF  CPAF  CPFF  T&M/LaborHonr 

Contract  Type  Continuum 

Figure  8:  Contract  Risk  Compared  With  PCE  Utility 

Source:  Developed  by  researcher. 
1 .    Firm  Fixed- Price 

Fixed-Price  contract  arrangements  of  all  types  represent 
the  majority  of  contracts  written.  [Ref  39:p.310]  In  1994, 
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these  contracts  made  up  75%  of  the  contracts  written  by  DoD. 
[Ref  45:p.81]  Some  use  was  the  median  descriptor  response  to 
the  question  of  the  utility  of  parametric  cost  estimating 
methods  to  Firm  Fixed-Price  (FFP)  type  contracts.  This  answer 
was  chosen  by  60  participants  in  the  survey  while  55  chose  the 
very  useful  descriptor  to  describe  the  utility.  Analysis  of 
variance  of  the  DoD  components  supports  this  opinion  of  the 
utility  of  parametric  cost  estimating  in  FFP  contract 
situations.  Table  24  contains  group  data  for  this  question 
and  Table  25  contains  the  analysis  of  variance  data. 


Table  24 
FFP  Contract  Group  Data 


Min 

Max 

M 

Median 

a 

Ql 

Q3 

Control 
Group 

1.00 

5.00 

2.00 

2.00 

1.32 

1.00 

2.50 

Program 
Managers 

1.00 

3.00 

2.33 

2.00 

.82 

1.75 

3.00 

Industry 

1.00 

5.00 

2.05 

2.00 

1.13 

1.00 

3.00 

Contracting 
Officers 

1.00 

4.00 

2.58 

3.00 

.87 

2.00 

3.00 

Source:  Developed  by  researcher. 

The  U.S.  Army,  DCAA,  U.S.  Navy,  and  USAF  groups,  totaling 
105  respondents,  had  average  response  values  significantly 
higher  than  the  average.  The  groups  identified  in  Table  24, 
however,  make  decisions  about  which  type  of  contract  to  use  in 
situations  that  they  are  presented  with  and  thus  may  have  more 
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influence  in  this  area.  Although  there  is  not  a  consensus 
across  the  groups  or  within  any  one  group,  with  the  exception 
of  the  contracting  officers,  the  group  averages  are  somewhat 
lower  than  the  average  for  the  entire  database  (2.56) .  This 
demonstrates  a  view  that  the  methods  are  very  useful  in  FFP 
contracts . 


Table  25 
Analysis  of  Variance  For  FFP  Contract  Data 


DoD  Component 

DF 

sd2 

O2 

F 

P 

Between  Groups 

6 

9.99 

1.66 

2.31 

.04 

Within  Groups 

130 

93.77 

.72 

Source:  Developed  by  researcher. 

2.    Fixed-Price  With  Economic  Price  Adjustment 

Responses  to  the  question  of  the  utility  of  Fixed-Price 
With  Economic  Price  Adjustment  (FPE)  type  contracts  was 
primarily  that  they  had  some  use.  The  average  response  value 
was  2.71  and  there  was  a  median  value  of  3  to  this  question. 
There  were  also  a  large  number  (81)  of  no  responses.  The  USAF 
respondents  had  a  consensus  that  they  were  only  of  some  use  in 
these  contract  situations.  Data  from  the  group  analysis  is 
presented  in  Table  26.  The  benchmark  group  responses  reflect 
a  higher  regard  for  the  use  of  the  methods  in  situations  where 
one  would  use  FPE  contracts  than  did  the  entire  pool  of 
participants . 
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Table  26 
FPE  Contract  Group  Data 


Min 

Max 

M 

Median 

a 

Ql 

Q3 

Control 
Group 

1.00 

5.00 

2.14 

2.00 

1.46 

1.00 

3.00 

Program 
Managers 

1.00 

3.00 

2.25 

2.50 

.96 

1.25 

3.00 

Industry 

1.00 

5.00 

2.00 

2.00 

1.19 

1.00 

3.00 

Contracting 
Officers 

1.00 

4.00 

2.77 

3.00 

.81 

2.00 

3.00 

USAF 

2.00 

4.00 

2.93 

3.00 

.62 

2.75 

3.00 

Source:  Developed  by  researcher. 

3.    Fixed-Price  Incentive,  Successive  Target 

The  aggregate  group  response  to  this  question  was  one  of 
some  use  for  the  methods  in  situations  that  a  Fixed-Price 
Incentive,  Successive  Target  (FPIS)  contract  arrangement  was 
suitable.  This  question  had  a  large  number  (83)  of  no 
responses  but  of  those  who  did  answer,  59  selected  some  use, 
and  42  selected  very  useful  as  their  answers.  This  produced 
an  average  response  value  of  2.62  and  the  median  of  3.  Aside 
from  the  benchmark  groups,  the  only  group  with  any  type  of 
consensus  on  this  question  was  the  Government  policy  group. 
These  participants  also  felt  strongly  that  the  methods  were 
only  somewhat  useful  in  FPIS  situations.  The  contracting 
officers  also  echoed  this  belief  while  the  other  benchmark 
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groups  felt  that  the  methods  were  very  useful  in  these 
contract  situations.   Table  27  presents  this  information. 


Table  27 
FPIS  Contract  Group  Data 


Min 

Max 

M 

Median 

a 

Ql 

Q3 

Control  Group 

1.00 

3.00 

1.83 

1.50 

.98 

1.00 

3.00 

Program 
Managers 

1.00 

3.00 

2.20 

2.00 

.84 

1.50 

3.00 

Industry 

1.00 

4.00 

1.95 

2.00 

.97 

1.00 

3.00 

Government 
Policy 

2.00 

4.00 

2.81 

3.00 

.68 

2.00 

3.00 

Contracting 
Officers 

2.00 

4.00 

2.72 

3.00 

.67 

2.00 

3.00 

Source:  Developed  by  researcher. 

4.    Fixed-Price  With  Prospective  Redetermination 

Parametric  cost  estimating  methods  have  essentially  the 
same  utility  value  in  Fixed-Price  With  Prospective 
Redetermination  (FPR(P))  contracts  as  they  do  in  FPIS  contract 
situations.  The  aggregate  response  average  was  2.60  with  the 
median  value  equal  to  3  or  some  use.  Again  there  were  a  large 
number  of  no  responses  (77)  and  the  distribution  was 
positively  skewed.  Analysis  of  variance  showed  that  the  NASA 
respondents  thought  that  the  methods  were  more  useful  than  did 
the  DoD  component  groups.  The  data  are  presented  in  Tables  28 
and  29. 
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Table  28 
FPR(P)  Contract  Group  Data 


Min 

Max 

V- 

Median 

a 

Ql 

Q3 

Control  Group 

1.00 

3.00 

1.83 

1.50 

.98 

1.00 

3.00 

Program 
Managers 

1.00 

3.00 

2.20 

2.00 

.84 

1.50 

3.00 

Industry- 

1.00 

4.00 

1.90 

2.00 

.94 

1.00 

3.00 

Contracting 
Officers 

2.00 

4.00 

2.74 

3.00 

.67 

2.00 

3.00 

Government 
Policy 

1.00 

4.00 

2.68 

3.00 

.78 

2.00 

3.00 

NASA 

1.00 

2.00 

1.67 

2.00 

.59 

1.00 

2.00 

Source:  Developed  by  researcher. 


Table  29 
Analysis  of  Variance  For  FPR(P)  Contract  Data 


DoD  Component 

DF 

2d2 

O2 

F 

P 

Between  Groups 

6 

8.32 

1.39 

2.69 

.02 

Within  Groups 

109 

56.19 

.52 

Source:  Developed  by  researcher. 

Government  policy  and  contracting  officer  groups  differ 
in  their  opinion  of  the  utility  of  the  methods  to  these  type 
of  contract  situations.  Both  of  these  groups  believe  that 
parametric  cost  estimating  methods  have  limited  use  in  FPR(P) 
contract  situations  while  the  remaining  groups  in  Table  28 
believe  that  they  are  very  useful. 
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5.    Fixed-Ceiling-Price      With     Retroactive 
Redetermination 

The  results  from  the  analysis  of  Fixed-Ceiling-Price  With 
Retroactive  Redetermination  FPR(R)  contract  type  question 
mirror  those  of  the  FPR(P)  question.  The  composite  group 
response  average  was  2.64  with  the  median  value  equal  to  3  or 
some  use.  There  were  a  large  number  of  no  responses  (79)  and 
the  distribution  was  positively  skewed.  The  control,  program 
manager,  and  industry  groups  thought  that  the  methods  were 
very  useful  in  these  types  of  contract  situations  as  did  the 
NASA  participants.     These  data  are  presented  in  Table  30. 

The  analysis  of  variance  for  DoD  component  and  job 
classification,  presented  in  Table  31,  highlighted  not  only 
the  NASA  group  but  also  the  cost  estimators,  auditors,  and 
cost  and  price  analysts.  The  last  three  groups  listed 
believed,  as  do  the  contracting  officers,  that  parametric  cost 
estimating  methods  have  only  some  use  in  FPR(R)  contracting 
arrangements.  The  opinion  of  these  groups  is  significant. 
They  represent  a  major  part  (107)  of  the  participants  and  a 
cross  section  of  acquisition  community  who  might  be  expected 
to  be  involved  in  applying  the  methods  to  a  FPR(R)  situation. 
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Table  30 
FPR(R)  Contract  Group  Data 


Min 

Max 

M 

Median 

a 

Ql 

Q3 

Control 
Group 

1.00 

3.00 

1.67 

1.50 

.82 

1.00 

2.25 

Program 
Managers 

1.00 

3.00 

2.00 

2.00 

.71 

1 .  50 

2.50 

Industry- 

1.00 

4.00 

1.84 

2.00 

.96 

1.00 

3.00 

Contracting 
Officers 

2.00 

4.00 

2.76 

3.00 

.71 

2.00 

3.00 

NASA 

1.00 

2.00 

1.67 

2.00 

.58 

1.00 

2.00 

Cost 
Estimators 

1.00 

4.00 

2.67 

2.00 

1.21 

1.75 

4.00 

Cost  or 

Price 

Analysts 

1.00 

4.00 

2.71 

3.00 

.90 

2.00 

3.00 

Auditors 

2.00 

4.00 

2.80 

3.00 

.71 

2.00 

3.00 

Source:  Developed  by  researcher 


Table  31 
Analysis  of  Variance  For  FPR(R)  Contract  Data 


Job  Classification 

DF 

Sd2 

G2 

F 

P 

Between  Groups 

9 

14.12 

1.57 

2.41 

.02 

Within  Groups 

118 

76.76 

.65 

DoD  Component 

DF 

sd2 

a2 

F 

P 

Between  Groups 

6 

8.87 

1.48 

2.62 

.02 

Within  Groups 

108 

61.08 

.57 

Source:  Developed  by  researcher. 
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6.    Fixed-Price-Incentive,  Firm 

Group  responses  to  this  question  were  almost  evenly 
divided  between  some  use  (57)  and  very  useful  (52).  The 
median  response  was  some  use  and  the  mean  response  value  was 
2.56.  Tables  32  and  33  show  the  information  for  this 
question. 


Table  32 
FPIF  Contract  Group  Data 


Min 

Max 

M 

Median 

a 

Ql 

Q3 

Control 
Group 

1.00 

3.00 

1.63 

1.50 

.74 

1.00 

2.00 

Program 
Managers 

1.00 

3.00 

2.33 

2.00 

.82 

1.50 

2.50 

Industry 

1.00 

4.00 

1.90 

2.00 

.97 

1.00 

3.00 

Contracting 
Officers 

2.00 

4.00 

2.64 

3.00 

.68 

2.00 

3.00 

Cost  or 

Price 

Analysts 

1.00 

4.00 

2.64 

3.00 

.90 

2.00 

3.00 

Auditors 

2.00 

4.00 

2.71 

3.00 

.69 

2.00 

3.00 

Source:  Developed  by  researcher. 

The  control  and  industry  groups  felt  that  the  methods  had 
more  utility  for  Fixed-Price-Incentive,  Firm  (FPIF)  contracts 
than  the  other  groups  that  were  surveyed.  These  two  groups 
indicated  that  the  methods  were  very  useful  while  the 
remaining  groups  felt  that  they  had  only  limited 
applicability.  The  analysis  of  variance  data  indicate  that 
cost  and  price  analysts,  contracting  officers,  and  auditors 
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(99   of   the   respondents)   thought   that   parametric   cost 
estimating  methods  had  only  some  use. 


Table  33 
Analysis  of  Variance  For  FPIF  Contract  Data 


Job  Classification 

DF 

Zd2 

a2 

F 

P 

Between  Groups 

9 

16.98 

1.89 

2.97 

.00 

Within  Groups 

127 

80.61 

.64 

Source:  Developed  by  researcher. 
7.    Cost-Pius-Incentive-Fee 

Cost-Reimbursement  type  contracts  formed  a  significant 
part  (18%)  of  the  contracts  awarded  by  the  DoD  in  1994.  [Ref 
45:p.81]  Parametric  cost  estimating  method  utility  in  Cost- 
Plus-Incentive-Fee  (CPIF)  contract  situations  was  determined 
to  be  somewhat  useful  by  the  survey  respondents.  An  average 
response  value  of  2.62  and  median  of  3  was  observed.  Contract 
negotiators,  program  managers,  industry  respondents,  and 
engineers  (37  participants  total) ,  identified  in  the  analysis 
of  variance  data,  thought  that  the  methods  would  be  very 
useful  in  CPIF  contract  situations.  The  remainder  of  the 
respondents  (103) ,  represented  in  Table  34  by  the  contracting 
officer  group,  thought  that  the  methods  were  of  only  some  use 
in  this  type  of  situation.  It  is  of  interest  to  note  the 
difference  of  opinion  regarding  the  utility  of  parametric  cost 
estimating  methods  in  CPIF  situations  between  the  contracting 
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officers  and  contract  negotiators.  One  would  expect  to  find 
similar  responses  to  this  question  from  these  two  groups 
because  of  their  common  training  and  experience.  Analysis  of 
variance  data  are  presented  in  Table  35. 


Table  34 
CPIF  Contract  Group  Data 


Min 

Max 

M 

Median 

a 

Ql 

Q3 

Control 
Group 

1.00 

2.00 

1.50 

1.50 

.54 

1.00 

2.00 

Program 
Managers 

1.00 

3.00 

1.80 

2.00 

.84 

1.00 

2.50 

Industry 

1.00 

4.00 

1.80 

1.00 

1.15 

1.00 

2.75 

Negotiators 

1.00 

2.00 

1.50 

1.50 

.58 

1.00 

2.00 

Engineers 

1.00 

2.00 

1.63 

2.00 

.52 

1.00 

2.00 

Contracting 
Officers 

1.00 

4.00 

2.71 

3.00 

.80 

2.00 

3.00 

Source:  Developed  by  researcher 


Table  35 
Analysis  of  Variance  For  CPIF  Contract  Data 


Job  Classification 

DF 

2d2 

O2 

F 

P 

Between  Groups 

9 

21.03 

2.34 

3.41 

.00 

Within  Groups 

131 

89.79 

.69 

Source:  Developed  by  researcher. 
8.    Cost-Pius -Award-Fee 

The  median  value  of  3  reflects  the  choice  of  some  use  by 
55  participants  to  describe  the  utility  of  parametric  cost 
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estimating  methods  to  Cost-Plus-Award-Fee  (CPAF)  contract 
situations.  The  average  response  value  was  slightly  lower  at 
2.59,  in  large  part  due  to  43  respondents  selecting  very- 
useful  as  their  answer  to  the  question.  The  analysis  of 
variance  data  showed  a  clear  separation  of  opinion  regarding 
the  utility  of  the  methods  to  CPAF  contract  situations. 
Contract  negotiators,  program  managers,  and  engineers  (17) 
answered  that  they  were  very  useful  and  all  of  the  remaining 
job  classification  groups  (123)  had  the  opinion  that  they  were 
only  somewhat  useful. 

The  analysis  of  variance  information  from  the  DoD 
component  groups  showed  that  the  USAF  had  the  lowest  response 
average  and  the  U.S.  Army  had  the  highest  average  in  those 
groups.  The  data  in  Table  36  show  that  the  USAF  indicates 
that  the  methods  are  very  useful  and  the  U.S.  Army  only  of 
some  use  in  CPAF  contract  situations.  Analysis  of  variance 
information  is  presented  in  Table  37 .  The  divergence  between 
contracting  officer  and  contract  negotiator  opinions  regarding 
this  contract  type  and  parametric  cost  estimate  utility 
observed  in  earlier  questions  is  also  noted  here.  Also 
noteworthy  is  the  consensus  that  the  control  and  engineer 
groups  have  on  the  question.  They  both  believe  that  the 
methods  are  very  useful  or  vital  in  a  situation  where  one 
would  use  this  type  of  contract. 
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Table  36 
CPAF  Contract  Group  Data 


Min 

Max 

M 

Median 

a 

Ql 

Q3 

Control 
Group 

1.00 

2.00 

1.50 

1.50 

.54 

1.00 

2.00 

Program 
Managers 

1.00 

3.00 

1.80 

2.00 

.84 

1.00 

2.50 

Industry- 

1.00 

4.00 

1.80 

1.00 

1.15 

1.00 

2.75 

Engineers 

1.00 

2.00 

1.57 

2.00 

.54 

1.00 

2.00 

Contract 
Negotiators 

1.00 

3.00 

1.80 

2.00 

.84 

1.00 

2.50 

U.S.  Army 

1.00 

4.00 

2.82 

3.00 

.83 

2.00 

3.25 

USAF 

1.00 

4.00 

2.13 

2.00 

.99 

1.00 

3.00 

Contracting 
Officers 

1.00 

4.00 

2.72 

3.00 

.78 

2.00 

3.00 

Source:  Developed  by  researcher 


Table  37 
Analysis  of  Variance  For  CPAF  Contract  Data 


Job  Classification 

DF 

Sd2 

a2 

F 

P 

Between  Groups 

9 

20.26 

2.25 

2.93 

.00 

Within  Groups 

129 

98.98 

.77 

DoD  Category 

DF 

Ed2 

a2 

F 

P 

Between  Groups 

6 

13.76 

2.29 

3.06 

.01 

Within  Groups 

117 

87.69 

.75 

Source:  Developed  by  researcher 
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9. 


Cost-Pius -Fixed-Fee 


There  were  no  groups  that  displayed  a  consensus  or 

unanimous  opinion  regarding  the  use  of  parametric  cost 

estimating  methods  in  situations  that  would  also  benefit  from 

the  use  of  a  Cost-Plus-Fixed-Fee  (CPFF)  arrangement.   Of  the 

benchmark  groups,  only  the  control  and  contracting  officer 

groups  had  a  consensus  of  some  sort  on  this  question.   Table 

38  contains  only  the  question  three  benchmark  groups  for 

analysis. 

Table  38 
CPFF  Contract  Group  Data 


Min 

Max 

M 

Median 

a 

Ql 

Q3 

Control 
Group 

1.00 

2.00 

1.38 

1.00 

.52 

1.00 

2.00 

Program 
Managers 

1.00 

3.00 

1.80 

2.00 

.84 

1.00 

2.50 

Industry 

1.00 

4.00 

1.65 

1.00 

1.13 

1.00 

2.00 

Contracting 
Officers 

2.00 

4.00 

2.69 

3.00 

.79 

2.00 

3.00 

Source:  Developed  by  researcher. 

The  aggregate  group  average  response  to  this  question  was 
2.58  and  a  median  response  of  somewhat  useful  was  recorded. 
The  control  group  had  the  highest  regard  for  using  the  methods 
in  CPFF  contract  situations,  very  useful  or  vital,  and  the 
contracting  officers  the  lowest  regard  for  their  use  in  CPFF 
instances.  In  comparison  with  the  average  value  for  the 
entire  database,  the  control,  program  manager,  and  industry 
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groups  had  a  much  higher  regard  for  the  utility  of  parametric 
cost  estimating  techniques  in  CPFF  contracting  than  did  any 
other  group. 

10.   Time  &  Materials 

While  Time  &  Materials  (T&M)  contracts  make  up  only  a 
small  percentage  (6%  in  FY  94)  of  the  contracts  awarded  by  the 
DoD,  a  significant  dollar  value  is  contracted  using  this 
vehicle.  [Ref  45:p.81]  The  majority  of  the  responses  to  this 
question  were  divided  between  some  use  and  no  use  in  an  area 
that  a  T&M  contract  might  also  be  beneficial.  The  largest 
number  of  respondents  chose  some  use  (53)  as  their  answer  and 
49  participants  chose  4,  or  no  use  to  the  question.  The 
average  response  value  of  3.13  and  median  response  of  3 
indicates  that  the  participants  felt  that  the  methods  were  of 
limited  use  in  these  situations.  A  relatively  insignificant 
number  (6)  of  people  chose  to  respond  that  the  methods  would 
be  a  hindrance  in  situations  requiring  a  T&M  contract. 

The  benchmark  group  data  and  groups  identified  during  the 
analysis  of  variance  are  shown  in  Table  39.  Table  40  contains 
analysis  of  variance  results  for  review.  With  the  exception 
of  the  program  managers,  there  is  an  almost  universal  belief 
that  the  methods  have  little  application  in  T&M  contracting 
situations.  Program  staff  opinion  differs  from  the  program 
managers  in  their  belief  about  the  utility  of  the  methods 
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here.  Program  staff  respondents  are  unanimous  in  their 
opinion  that  parametric  cost  estimating  methods  are  of  no  use 
in  T&M  contract  situations  while  program  managers  indicate 
that  they  believe  that  they  have  some  use  in  the  process.  The 
groups  shown  in  Table  39  show  a  belief  throughout  the  DoD 
acquisition  community  that  the  methods  are  of  little  use  in 
T&M  situations. 


Table  39 
T&M  Contract  Group  Data 


Min 

Max 

M 

Median 

a 

Ql 

Q3 

Control  Group 

1.00 

5.00 

3.29 

3.00 

1.50 

2.00 

5.00 

Program 
Managers 

1.00 

4.00 

2.33 

2.50 

1.21 

1.00 

3.25 

Program  Staff 

4.00 

4.00 

4.00 

4.00 

.00 

4.00 

4.00 

Industry 

1.00 

5.00 

2.90 

3.00 

1.37 

2.00 

4.00 

Cost  or  Price 
Analysts 

1.00 

5.00 

3.27 

3.00 

1.03 

3.00 

4.00 

Contracting 
Officers 

2.00 

5.00 

3.15 

3.00 

.82 

2.75 

4.00 

Source:  Developed  by  researcher. 


Table  40 
Analysis  of  Variance  For  T&M  Contract  Data 


Job  Classification 

DF 

Sd2 

a2 

F 

P 

Between  Groups 

9 

17.21 

1.91 

2.31 

.02 

Within  Groups 

128 

105.89 

.83 

Source:  Developed  by  researcher 
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1 1 .   Labor  Hour 

The  results  of  the  analysis  of  the  Labor  Hour  contracts, 
representing  1%  of  all  DoD  procurement  actions,  are  similar  to 
those  of  the  T&M  contracts.  [Ref  45:p.81]  Again,  there  was  a 
median  response  of  only  some  use  and  an  average  value  of  3.13. 
Responses  from  the  benchmark  groups  in  Table  41  reflect  a  view 
that  the  methods  have  little  application  in  this  type  of 
contract  environment.  The  program  staff  group  is  unanimous  in 
their  opinion  that  the  methods  have  no  use  in  labor  hour 
contract  type  situations.  This  differs  slightly  from  the  view 
of  the  program  managers  who  see  some  application  for 
parametric  cost  estimating  methods  in  labor  hour  contracts. 


Table  41 
Labor  Hour  Contract  Group  Data 


Min 

Max 

M 

Median 

a 

Ql 

Q3 

Control 
Group 

1.00 

5.00 

3.50 

4.00 

1.76 

1.75 

5.00 

Program 
Managers 

2.00 

4.00 

3.00 

3.00 

.71 

2.50 

3.50 

Program 
Staff 

4.00 

4.00 

4.00 

4.00 

.00 

4.00 

4.00 

Industry 

1.00 

5.00 

3.11 

3.00 

1.28 

2.00 

4.00 

Contracting 
Officers 

2.00 

5.00 

3.11 

3.00 

.87 

2.00 

4.00 

Source:  Developed  by  researcher. 
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The  analysis  of  the  various  contract  types  indicates  that 
there  are  some  contract  types  that  can  benefit  through 
applying  parametric  cost  estimating  methods  to  the  process 
more  than  others.  The  average  values  for  the  control ,  program 
manager,  and  industry  groups  indicate  a  relationship  between 
risk,  contract  type,  and  the  utility  of  parametric  cost 
estimating  methods  similar  to  that  shown  in  Figure  8. 
Respondents  in  these  groups  viewed  the  methods  as  having  the 
most  applicability  in  CPFF  and  least  utility  with  T&M  or  labor 
hour  contract  situations.  While  this  relationship  exists,  it 
may  not  be  as  pronounced  as  that  shown  in  Figure  8 . 
Furthermore,  the  relationship  shows  a  distribution  of  utility 
that  is  skewed  to  the  right.  Figure  9  graphically  displays 
the  average  values  for  the  various  contract  types. 
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Figure  9:  Contract  Type  Compared  to  Parametric  Cost  Estimating  Utility 

Source:  Developed  by  researcher. 
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The  increase  observed  in  the  utility  lines  for  the 
control,  program  manager,  and  industry  groups  is  not  present 
in  the  contracting  officer  line.  The  contracting  officers  may 
view  risk  from  an  individual  contract  standpoint  rather  than 
across  an  entire  program.  They  may  see  parametric  cost 
estimating  as  a  tool  to  be  evaluated  for  use  in  one  single 
action  and  not  having  any  relationship  to  risks  in  the 
contract  or  program.  This  group  is  also  made  up  of  a 
combination  of  procuring  contracting  officers  (PCO)  as  well  as 
administrative  contracting  officers  (ACO) .  These  two 
subgroups  may  have  different  viewpoints  on  the  utility  of  the 
methods  based  on  risk  and  contract  type  that  are  masked  when 
they  are  combined  into  one  group.  These  differences  could  be 
a  result  of  the  roles  both  groups  have  in  the  acquisition 
process  and  program  management  cycles. 

All  of  the  benchmark  groups  for  this  question  responded 
that  there  was  little  or  no  application  for  parametric  cost 
estimating  methods  in  either  T&M  or  labor  hour  contract 
situations.  This  type  of  contract  is  usually  used  in 
situations  where  the  work  can  be  described  but  not  the 
quantity  of  work  needed.  Pricing  for  the  labor  or  materials 
is  performed  by  determining  actual  labor  rates  or  material 
costs  based  on  the  marketplace.  These  two  factors  combine  to 
make  parametric  cost  estimating  methods  less  efficient  in 
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these  situations  than  other  estimating  methods.   However,  the 
methods  should  be  able  to  provide  some  use  if  adequate 
databases  and  analysis  techniques  are  available  to  project  the 
required  quantities  of  labor  or  materials. 
D.     PRE -AWARD  &  POST-AWARD  CONTRACT  ACTIONS 

Question  four  asks  the  participants  to  offer  an  opinion 
on  the  utility  of  parametric  cost  estimating  methods  to  either 
pre-award  actions  or  post-award  actions.  Pre-award  activities 
where  parametric  cost  estimating  might  be  useful  include 
budgeting,  ICE  generation,  proposal  analysis,  or  preparing  for 
negotiations  and  price  analysis.  Post-award  actions  in  which 
the  methods  may  be  useful  could  include  evaluating  an 
Engineering  Change  Proposal  (ECP) ,  analyzing  the  cost  aspects 
of  a  claim,  negotiating  a  Forward  Pricing  Rate  Agreement,  or 
negotiating  changes  to  existing  contracts.  These  actions 
could  be  performed  by  either  PCO  or  ACO  personnel  at  various 
points  in  the  acquisition  process. 

1 .    Pre-Award  Contract  Actions 

Pre-award  functions  such  as  those  previously  described 
are  the  areas  in  which  parametric  cost  estimating  methods  are 
usually  thought  to  apply  most  directly.  Given  this  pre- 
disposition it  is  not  surprising  to  find  that  the  responses 
show  a  relatively  high  degree  of  utility  for  the  methods  in 
pre-award  contract  actions.   The  average  response  value  for 
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this  question  is  2.36  and  the  median  value  is  2  or  very 
useful.  All  of  the  responses  were  coded  4  or  lower  indicating 
that  no  one  felt  that  the  methods  were  a  hindrance  in  pre- 
award  actions.  Also,  only  13  responses  were  coded  as  no  use 
for  the  methods  in  pre-award  applications.  Tables  42  and  43 
present  the  group  data  and  analysis  of  variance  data  for  this 
question. 


Table  42 
Pre-Award  Contract  Action  Group  Data 


Min 

Max 

M 

Median 

a 

Ql 

Q3 

Program 
Staff 

3.00 

3.00 

3.00 

3.00 

.00 

3.00 

3.00 

Control 
Group 

1.00 

1.00 

1.00 

1.00 

.00 

1.00 

1.00 

Corporate 
Policy 

1.00 

2.00 

1.60 

2.00 

.55 

1.00 

1.50 

Program 
Managers 

1.00 

3.00 

1.57 

1.00 

.79 

1.00 

2.00 

Source:  Developed  by  researcher 


Table  43 
Analysis  of  Variance  For  Pre-Award  Action  Data 


Job  Classification 

DF 

Sd2 

O2 

F 

P 

Between  Groups 

9 

12.16 

1.35 

1.94 

.05 

Within  Groups 

161 

111.88 

.90 

Source:  Developed  by  researcher. 
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All   of   the   job   classification   groups   think   that 

parametric  cost  estimating  methods  are  useful  in  pre-award 

actions  but  the  analysis  of  variance  data  highlighted  the 

program  staff  and  program  manager  groups.   These  two  groups 

have  a  significant  difference  of  opinion  regarding  the  utility 

of  the  methods.   The  program  managers  believe  that  parametric 

cost  estimating  methods  are  vital  or  very  useful  in  pre-award 

situations  while  the  program  staff  are  unanimous  in  their 

belief  that  the  methods  are  only  of  some  use.   The  control 

group  also  is  unanimous  in  their  opinion.   They  feel  that  the 

methods  are  vital  to  pre-award  actions.   Another  group  of 

interest,  the  corporate  policy  group,  feels  that  parametric 

cost  estimating  methods  are  very  useful  in  these  actions. 

This  is  also  represented  in  the  regression  equation  that 

relates  business  size  to  the  responses.   This  equation 

Y  =  .351 (X)  -  6.56         R2  =  .21 
(.15)     (2.03) 

indicates  that  there  is  a  negative  relationship  between 

business  size  and  the  response  to  this  question.   As  business 

size  increases  the  respondents  believe  that  the  methods  are 

more  applicable  in  pre-award  contract  actions.   This  might  be 

expected  as  the  larger  businesses  have  more  experience  with 

using  the  methods  than  the  smaller  ones  do.   With  this  strong 

response,  the  use  of  parametric  cost  estimating  methods  should 

continue  to  be  used  extensively  in  pre-award  contract  actions. 
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2 .    Post-Award  Contract  Actions 

Parametric  cost  estimating  methods  are  not  usually 
associated  with  post-award  contract  actions.  Often  other  cost 
or  pricing  analysis  techniques  based  on  costs  experienced  up 
to  that  point  in  the  contract  are  used  to  price  changes, 
modifications,  claims,  or  other  post-award  actions.  The 
responses  to  this  question  reflect  a  view  that  the  methods 
have  little  use  in  post-award  contract  actions.  The  answers 
were  almost  equally  divided  between  some  use  (59)  and  no  use 
(53)  producing  an  average  response  value  of  3.04  and  median 
response  of  some  use.  A  relatively  insignificant  number  (8) 
of  respondents  felt  that  the  methods  could  actually  hinder 
post-award  contract  actions. 

Analysis  of  variance  data  showed  a  relationship  between 
the  responses  and  the  DoD  component  and  job  classification 
groups.  These  data  are  presented  in  Table  45.  Group  data  are 
presented  in  Table  44.  From  the  analysis  of  variance,  it  was 
noted  that  the  control  group  found  the  methods  to  be  vital  or 
very  useful  in  post-award  contract  actions  as  well  as  in  pre- 
award  contract  actions.  Program  managers  and  NASA  respondents 
agreed  with  the  control  group  in  believing  that  the  methods 
were  very  useful  in  these  activities.  The  DCAA  participants 
felt  that  parametric  cost  estimating  had  little  if  any  utility 
in  post-award  contract  actions. 
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The  NASA  response  and  their  long  history  of  involvement 
with  parametric  cost  estimating  offers  the  potential  for 
greater  use  in  this  area.  This  group  believes  that  the 
methods  are  applicable  in  certain  post-award  contract  actions 
as  do  the  program  managers  and  the  control  group.  The  methods 
should  provide  the  same  benefits  to  a  post-award  negotiation 
or  acquisition  plan  review  that  they  do  in  preparing  for 
negotiations  in  the  pre-award  phase  or  in  drafting  an 
acquisition  plan. 


Table  44 
Post-Award  Contract  Action  Group  Data 


Min 

Max 

M 

Median 

a 

Ql 

Q3 

Control 
Group 

1.00 

2.00 

1.23 

1.00 

.35 

1.00 

1.00 

Program 
Managers 

1.00 

3.00 

2.14 

2.00 

.90 

1.00 

3.00 

NASA 

1.00 

4.00 

2.50 

2.50 

.93 

1.00 

4.00 

DCAA 

2.00 

5.00 

3.75 

4.00 

.73 

3.00 

4.00 

Source:  DeveJ 

.oped  fc 

y  rese 

archer. 

The  DCAA  group  response  has  the  highest  value  of  any  of 
the  DoD  component  or  job  classification  groups.  Where  the 
program  managers  are  at  least  one  level  removed  from 
documenting  a  post-award  agreement,  the  DCAA  and  contracting 
officer  groups  are  the  individuals  who  bring  it  to  fruition. 
Their  belief  that  the  methods  are  of  little  use  in  post-award 
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actions  is  a  strong  counter  to  the  control  and  program  manager 
groups'  drive  to  use  the  methods  in  post-award  contract 
activities. 


Table  45 
Analysis  of  Variance  For  Pre-Award  Action  Data 


Job  Classification 

DF 

2d2 

O2 

F 

P 

Between  Groups 

8 

29.63 

3.70 

3.96 

.00 

Within  Groups 

154 

143.95 

.94 

DoD  Component 

DF 

Zd2 

O2 

F 

P 

Between  Groups 

6 

29.60 

4.93 

5.81 

.00 

Within  Groups 

140 

118.80 

.85 

Source:  Developed  by  researcher. 

E.    SUMMARY 

Questions  one  through  four  attempted  to  bring  someone 

participating  in  the  survey  through  a  process  of  identifying 

that  person' s  underlying  beliefs  about  where  parametric  cost 

estimating  could  be  used.   The  opinions  presented  in  these 

guestions  address  the  basic  research  question  of: 

How  could  a  DoD  emphasis  on  using  parametric  methods  in 
cost  estimating  affect  the  procurement  process? 

The  responses  of  the  participants  form  the  foundation  of  the 
analysis  of  the  remaining  questions  as  they  address  specific 
subsidiary  research  questions. 
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The  remaining  questions,  five  through  ten,  ask  the 
respondent  to  address  issues  regarding  the  application  of  the 
methods  to  specific  contract  actions.  The  last  five  questions 
use  a  five  point  scale  of  agreement  or  disagreement  with  each 
statement.  A  response  of  one  or  two  indicates  agreement  and 
those  of  three  or  four  show  disagreement.  A  response  of  five 
is  coded  as  no  opinion. 
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VII.  ACQUISITION  AND  SOLICITATION  PLANNING 

A.  PURPOSE 

The  purpose  of  this  chapter  is  to  analyze  the  responses 
to  questions  five  and  six.  These  two  survey  questions  and  the 
first  four  interview  questions  address  planning  areas  that  may 
be  impacted  if  parametric  cost  estimating  methods  are  used. 
Questions  five  through  ten  used  a  five  point  scale  indicating 
the  degree  of  agreement  with  a  set  of  statements  within  each 
question  presented.  The  scale  ran  between  "strongly  agree, " 
a  one  on  the  scale,  to  "strongly  disagree, "  a  four  on  the 
scale,  with  a  five  on  the  scale  being  "no  opinion"'  regarding 
the  question.  For  questions  five  through  ten,  a  "no  response" 
was  coded  the  same  as  a  "no  opinion"  response  as  it  was 
entered  into  the  database. 

B.  METHODS,  MODELS,  AND  SOFTWARE 

Question  five  is  a  six  segment  series  of  statements  to 

which  the  respondents  indicated  that  they  agreed  or  disagreed. 

The  statements  are  directed  at  answering  the  first  subsidiary 

research  question, 

What  changes  could  offerors  make  to  their  proposal 
preparation,  submission  and  support  processes  to 
accommodate  using  parametric  methods,  techniques,  or 
software  in  estimating  potential  contract  costs? 

Responses  to  each  statement  will  be  analyzed  in  the  same 

manner  used  in  the  background  questions.   A  discussion  of  each 
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statement's  purpose  will  be  followed  by  group  data  analysis 
and  presentation  of  the  data. 

1.    Acquisition  Streamlining 

Question  5a  asks  respondents  to  provide  their  opinion 

regarding  the  following  statement: 

Parametric  cost  estimating  methods,  models,  and  software 
could  eventually  replace  traditional  bottoms  up  cost 
estimating  methods  in  many  acquisition  estimating 
applications  as  a  result  of  acquisition  streamlining. 

Many  acquisition  streamlining  initiatives  are  based  on  the 

principle  that  business  is  more  efficient  in  its  use  of 

resources  than  is  the  Federal  Government.   This  statement 

addresses  the  potential  displacement  of  traditional  estimating 

methods  by  parametric  cost  estimating  methods.   The  statement 

is  directed  at  determining  if  acquisition  streamlining  will 

accelerate  this  process.    The  effects  of  parametric  cost 

estimating  replacing  the  traditional  methods  are  addressed  in 

later  questions. 

The  aggregate  response  to  this  question  was  one  of 

indecision.   An  average  response  value  of  2.49  was  calculated 

from  a  response  distribution  of  1(28),  2(62),  3(67),  and 

4(28).   This  distribution  produced  a  median  value  of  3  or 

disagreement  with  the  statement.  The  program  staff  response, 

shown  in  Table  4  6,  is  very  clearly  disagreeing  with  the 

statement.   This  group  does  not  believe  that  traditional  cost 
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estimating  methods  will  be  supplanted  by  parametric  cost 
estimating  methods. 

There  are  other  groups  shown  in  Table  4  6,  however,  that 
do  believe  that  the  methods  could  eventually  replace 
traditional  methods  in  some  applications.  Engineers  and  those 
who  set  corporate  policy  believe  that  parametric  cost 
estimating  methods  could  be  used  in  many  more  applications  as 
the  primary  estimating  method.  Greater  discretion  given  to 
decision  makers  could  result  in  the  methods  being  applied  in 
more  situations. 


Table  46 
Question  5a  Group  Data 


Min 

Max 

V- 

Median 

a 

Ql 

Q3 

Program 
Staff 

3.00 

4.00 

3.75 

4.00 

.50 

3.25 

4.00 

Engineers 

1.00 

3.00 

2.00 

2.00 

.63 

2.00 

2.00 

Corporate 
Policy 

1.00 

2.00 

1.40 

1.00 

.55 

1.00 

2.00 

Source:  Developed  by  researcher. 

Figure  10  depicts  a  situation  where  a  precise  but  time- 
and-cost-intensive  detailed  estimate  and  a  somewhat  less 
precise  but  quicker  and  less  costly  parametric  estimate  are 
compared  at  the  same  point  in  the  acquisition  cycle.  [Ref  40] 
Both  estimates  predict  the  cost  of  the  equipment  will  cost 
between  $79  and  $140.   The  detailed  estimate  provides  a  higher 
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probability  that  it  will  cost  $105  than  any  other  price  in  the 
estimate  range.  In  some  situations,  the  use  of  good  business 
judgment  will  require  a  program  manager  or  contracting  officer 
to  use  the  less  precise  estimate. 

In  the  past  the  acquisition  community  has  relied  heavily 
on  precise,  detailed  estimates  often  contributing  to  program 
delays  and  added  costs.  Encouraging  acquisition  officials  to 
make  these  types  of  decisions  based  on  sound  business 
practices  is  a  critical  part  of  acquisition  streamlining. 
Parametric  cost  estimating  is  one  of  the  tools  available  to 
analyze  situations  and  make  decisions  based  on  good  business 
judgment. 
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Detailed  Estimate 


Parametric  Estimate 
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Figure  10:  Estimate  Accuracies 

Source:  Shields,  October  1996 
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2 .    Information  Systems  Improvements 

Through  statement  5b,  the  survey  participants  are  asked 
to  consider  how  technology  might  affect  the  cost  estimating 
methods  that  are  being  used. 


Parametric  cost  estimating  methods,  models,  and  software 
could  replace  traditional  cost  estimating  methods  as 
larger,  more  compatible  data  bases,  greater  computing 
power,  and  improved  software  become  available. 


Parametric  cost  estimating  methods  can  be  computer  asset 
intensive  and  improvements  in  either  hardware,  software,  or 
database  management  could  have  dramatic  effects  on  their 
precision.  The  participants  in  the  survey  agreed  with  the 
statement  that  improvements  in  computer  capability  could  allow 
parametric  cost  estimating  methods  to  supplant  more 
traditional  methods.  An  average  response  value  of  2.38  and 
median  response  of  agreement  or  2  were  observed  for  this 
question.  The  corporate  policy  group  was  unanimous  in  their 
agreement  to  this  statement.  They  had  an  average  response  of 
1.40  and  standard  deviation  of  .548  and  everyone  in  this  group 
agreed  or  strongly  agreed. 

Technology  advances  could  potentially  change  the 
characteristics  for  the  Figure  10  parametric  cost  estimate  to 
more  closely  match  those  of  the  detailed  estimate.  While  the 
performance  of  parametric  cost  estimating  systems  might 
improve  with  computer  technology,  advancements  there  might 
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also  benefit  traditional  methods.  The  same  increasing 
computer  capability  that  could  make  parametric  cost  estimating 
more  precise  could  also  be  used  to  make  bottoms  up  estimating 
more  accurate.  [Refs  20,30]  Greater  connectivity  between  data 
gathering  systems  could  reduce  the  cost  of  producing  a 
detailed  estimate  just  as  it  may  make  parametric  cost 
estimating  more  accurate  and  faster.  The  corporate  policy 
group  response  indicates  greater  acceptance  for  using 
parametric  cost  estimating  methods  in  future  applications, 
however. 

3 .    Advantages 

The  concepts  in  the  following  statement-  provide  the 

motivation   for   both   contractor   and   Federal   Government 

personnel  to  use  parametric  cost  estimating  methods  in  more 

areas: 

Parametric  cost  estimating  methods,  models,  and  software 
can  save  contractors  and  Federal  Government  oversight 
officials  significant  time  and  expense  in  developing  and 
then  evaluating  proposals. 

The  majority  (142)  of  the  survey  participants  agreed  with  this 
statement.  The  median  response  was  2  and  the  average  response 
value  observed  was  2.04.  The  group  data  presented  in  Table  47 
show  a  range  of  activities  across  the  acquisition  community 
that  believe  that  parametric  cost  estimating  methods  have  the 
potential  to  speed  up  the  acquisition  process  and  reduce  the 
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costs  of  proposal  preparation  and  evaluation.  The  control, 
engineer,  and  corporate  policy  groups  strongly  feel  that  these 
methods  can  provide  the  advantages  listed.  This  indicates 
that  there  is  a  willingness  to  use  the  methods  on  a  wider 
basis.  The  belief  and  acceptance  by  DCAA  of  the  potential 
that  the  methods  could  lead  to  greater  acceptance  for  using 
the  methods  in  more  areas  where  traditional  cost  estimating 
methods  have  been  used. 


Table  47 
Question  5c  Group  Data 


Min 

Max 

M 

Median 

a 

Qi 

Q3 

Control  Group 

1.00 

2.00 

1.27 

1.00 

.47 

1.00 

2.00 

Engineers 

1.00 

2.00 

1.45 

1.00 

.52 

1.00 

2.00 

Cost 
Estimators 

1.00 

3.00 

2.00 

2.00 

.77 

2.00 

2.00 

Corporate 
Policy 

1.00 

2.00 

1.20 

1.00 

.45 

1.00 

1.50 

DCAA 

1.00 

4.00 

2.15 

2.00 

.62 

2.00 

2.00 

Source:  Developed  by  researcher. 
4 .    Data  Collection 

Question  5d  asks  the  participants  to  record  their 

opinions  about  the  following  statement: 

Parametric  cost  estimating  methods,  models,  and  software 
could  significantly  change  the  way  potential  contractors 
collect  and  summarize  cost  information. 
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Many  cost  collection  systems  primarily  used  by  Federal 
Government  contractors  are  designed  or  have  been  modified  to 
collate  information  to  support  invoicing.  In  addition  to 
providing  management  information,  one  of  their  secondary 
functions  is  to  provide  data  for  their  cost,  estimating 
systems.  Data  available  from  these  systems  form  the  databases 
from  which  parametric  cost  estimates  may  be  produced.  This 
statement  addresses  the  concern  that  these  systems  may  not  be 
ideally  suited  for  producing  information  to  be  used  in 
parametric  cost  estimating  systems. 

The  participants  agreed  with  the  statement  that 
parametric  cost  estimating  could  affect  the  way  cost  data  are 
collected  and  presented.  A  median  value  of  2  and  an  average 
response  of  2.14  were  noted.  An  analysis  of  variance 
performed  on  the  DoD  component  response  data  showed  that  the 
NASA  and  USAF  groups  felt  more  strongly  in  agreement  with  this 
question  than  any  of  the  other  groups.  The  remaining  DoD 
component  groups  were  clustered  around  simple  agreement  with 
the  statement.  Within  the  USAF  group,  the  USAF  cost  analysts 
were  notable  for  their  strong  belief  that  the  methods  could 
change  the  way  cost  data  were  collected  and  summarized.  Group 
response  and  analysis  of  variance  data  are  presented  in  Tables 
4  8  and  49. 
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Table  48 
Question  5d  Group  Data 


Min 

Max 

M 

Median 

a 

Ql 

Q3 

USAF  Cost 
Analysts 

1.00 

2.00 

1.43 

1.00 

.54 

1.00 

2.00 

Corporate 
Policy 

1.00 

2.00 

1.20 

1.00 

.45 

1,00 

1.50 

USAF 

1.00 

3.00 

1.71 

2.00 

.59 

1.00 

2.00 

NASA 

1.00 

3.00 

1.50 

1.00 

.76 

1.00 

2.00 

Source:  Developed  by  researcher 


Table  49 
Analysis  of  Variance  For  Question  5d  Data 


DoD  Component 

DF 

2d2 

o2 

F 

P 

Between  Groups 

6 

12.98 

2.16 

4.05 

.00 

Within  Groups 

152 

81.17 

.53 

Source:  Developed  by  researcher. 
5.    Cost  Overruns 

Most  of  the  survey  participants  disagreed  with  the 

following  statement: 

Parametric  cost  estimating  methods,  models,  and  software 
should  decrease  the  probability  of  cost  overruns  where 
used. 

A  total  of  110  respondents  disagreed  with  the  statement  which 
produced  a  mean  response  value  of  2.89  and  disagree  (3) 
response  for  the  median.   The  analysis  of  variance  on  job 
classification    responses    showed    almost    overwhelming 
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disagreement  with  the  statement.   The  auditor  group  data  in 
Table  50  represents  that  view. 

Two  groups  in  the  analysis  of  variance  were  noteworthy 
because  of  their  agreement  with  the  statement.  Table  51 
contains  the  analysis  of  variance  data.  The  program  manager 
and  engineer  groups  feel  that  parametric  cost  estimating 
methods  may  have  some  use  in  preventing  cost  overruns. 


Table  50 
Question  5e  Group  Data 


Min 

Max 

M 

Median 

a 

Ql 

Q3 

Auditors 

2.00 

4.00 

3.97 

3.00 

.67 

2.75 

3.00 

Engineers 

1.00 

3.00 

2.00 

2.00 

.55 

2.00 

3.00 

College 
Algebra 

2.00 

4.00 

3.00 

3.00 

.66 

3.00 

3.00 

Program 
Managers 

1.00 

4.00 

1.86 

2.00 

1.07 

1.00 

2.00 

Source:  Developed  by  researcher. 


Table  51 
Analysis  of  Variance  For  Question  5e  Data 


Job  Classification 

DF 

Ed2 

a2 

F 

P 

Between  Groups 

9 

16.89 

1.88 

3.17 

.00 

Within  Groups 

145 

85.84 

.59 

Source:  Developed  by  researcher. 

There  is  also  a  relationship  between  mathematics 
education  and  the  responses  that  supports  the  group  data  in 
Table  50.   The  regression  equation 
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Y  =  -.094 (X)  +  6.40         R2  =  .02 
(.04)     (1.66) 

indicates  a  weak  negative  relationship  between  the  responses 

and  mathematics  education.   The  greater  level  of  mathematics 

education  that  both  groups  have  and  the  experience  that  the 

program  managers  have  may  have  produced  this  result.   These 

two  groups  may  view  the  issue  of  cost  in  terms  of  a  range  of 

possible  costs,  similar  to  that  presented  in  Figure  10,  that 

parametric  cost  estimating  methods  may  help  to  define.   Given 

the  flexibility  to  work  within  that  range  may  provide  a  way  to 

manage  risks  to  avoid  cost  overrun  situations. 

6.    Cost  and  Pricing  Data 

Given  the  response  to  statement  5d,  one  would  also  expect 

a  similar  response  to  the  following  statement: 

Parametric  cost  estimating  methods,  models,  and  software 
should  change  the  way  cost  and  pricing  data  submitted  to 
support  a  proposal  is  viewed. 

Cost  and  pricing  data  are  "...  all  the  facts  that  can  be 
reasonably  expected  to  contribute  to  the  soundness  of 
estimates  of  future  costs."  [Ref  13:part  15.801]  If  the 
methods  used  to  collect  and  summarize  data  are  changed  by  a 
potential  contractor  then  it  is  reasonable  to  assume  that  the 
cost  and  pricing  data  that  the  company  submits  to  the 
Government  in  a  proposal  will  also  change. 
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The  average  response  to  this  statement  was  agreement.  An 
aggregate  group  average  of  2.29  and  a  median  value  of  2  was 
noted.  A  broad  cross  section  of  the  DoD  acquisition  community 
is  presented  in  Table  52  who  agree  with  this  statement. 
Individuals  who  prepare  estimates  for  both  the  Federal 
Government  and  the  DoD  contractor  community  are  represented  in 
agreement  with  the  statement. 


Table  52 
Question  5f  Group  Data 


Min 

Max 

M 

Median 

a 

Ql 

Q3 

USAF 

Contracting 

Officers 

2.00 

3.00 

2.17 

2.00 

.41 

2.00 

2.25 

Negotiators 

2.00 

2.00 

2.00 

2.00 

.00 

2.00 

2.00 

Engineers 

1.00 

3.00 

2.00 

2.00 

.50 

2.00 

2.00 

Prepare 
Estimates 

1.00 

3.00 

1.97 

2.00 

.59 

2.00 

2.00 

USAF 

1.00 

4.00 

2.06 

2.00 

.77 

2.00 

2.00 

Source:  Developed  by  researcher. 

These  groups  represent  individuals  who  would  be  expected 
to  receive  and  then  analyze  a  proposal  to  a  solicitation  which 
could  include  cost  and  pricing  data.  All  of  the  groups  listed 
show  a  consensus  viewpoint.  The  contract  negotiator  group  is 
also  unanimous  in  their  agreement  with  the  statement. 

The  responses  to  this  series  of  statements  indicate  that 
the  commercial  industry  component  of  the  DoD  acquisition 
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community  is  willing  to  use  parametric  cost  estimating 
methods.  The  data  also  indicate  that  many  in  the  Federal 
Government  are  willing  to  accept  the  use  of  the  methods  also. 
To  take  this  step,  both  parties  should  agree  that  the 
contractor  community  may  have  to  take  several  actions. 
Generating  cost  estimates  based  on  parametric  methods  will 
require  investments  in  software  either  for  database  analysis 
and  the  generation  of  models  or  for  the  purchase  of  existing 
models  that  can  be  modified.  The  benefits  that  might  drive 
this  change  would  accrue  to  both  Government  and  industry  in 
the  reduced  acquisition  cycle  times,  lower  cost  proposal 
preparation  costs,  and  greater  management  flexibility.  It 
will  also  require  some  changes  to  the  systems  used  to  collect, 
collate,  and  analyze  cost  data  within  industry  and  the 
Government.  For  a  contractor  organization  that  elects  to  use 
parametric  cost  estimating  methods  as  a  primary  tool  for 
generating  data  for  contract  management,  significant  changes 
to  his  existing  structure  could  be  required. 
C.    SOLICITATION  PLANNING 

Question  6  addresses  the  issues  raised  in  the  second  and 
third  subsidiary  research  questions  in  seven  parts.  The 
questions  it  deals  with  are, 


1.  What  impacts  could  emphasis  on  using  cost  estimates 
based  on  parametric  cost  estimating  techniques  have  on 
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source  selection  and  on  the  evaluation  of  proposals  by 
the  DoD?  and 

2.  How  could  source  selection  criteria  and  evaluation 
factors  be  constructed  to  fairly  evaluate  a  proposal  that 
uses  a  parametric  estimate  with  one  that  does  not? 


Providing  a  level  playing  field  for  all  solicitation  proposals 

is  a  primary  goal  of  Federal  Government  contracting.   The 

statements  in  this  question  deal  with  how  the  Government  can 

construct  that  level  playing  field  within  the  solicitation  it 

issues  in  regard  to  the  use  of  parametric  cost  estimating 

methods. 

1.    Encouraging  Parametric  Cost  Estimating  Use 

The  first  statement  in  this  question  asks  the  respondent 

if   the   Government   should   simply   encourage   the   use   of 

parametric  cost  estimating  methods. 

In  solicitations,  buying  activities  should  encourage  the 
use  of  parametric  cost  estimating  for  specified 
procurement  actions. 

The  majority  of  respondents  overwhelmingly  agreed  with  this 
statement,  95  individuals  selected  agree,  and  27  selected 
strongly  agree.  An  average  response  value  of  2.21  and  a 
median  value  of  2  was  shown  in  the  cumulative  data.  The  group 
data  in  Table  53  is  a  display  of  groups  within  the  DoD 
acquisition  community  whose  responses  are  noteworthy  for  this 
question. 
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This  diverse  group  all  agree  that  the  proposing 
organization  should  be  encouraged  to  use  the  methods  where 
applicable.  As  one  might  expect,  the  group  representing 
industry  is  most  emphatic  in  their  agreement  with  this 
statement.  Given  the  present  environment  of  preference  for 
disengagement  from  intrusive  inspection  systems  and  burdensome 
requirements,  this  is  the  most  likely  form  of  emphasis  that 
DoD  will  use  to  promote  parametric  cost  estimating  techniques. 


Table  53 
Question  6a  Group  Data 


Min 

Max 

M 

Median 

a 

Ql 

Q3 

DCMC  Cost 
Analysts 

1.00 

3.00 

2.07 

2.00 

.59 

2.00 

2.00 

Negotiators 

1.00 

3.00 

2.00 

2.00 

.58 

2.00 

2.00 

Corporate 
Policy 

1.00 

2.00 

1.25 

1.00 

.50 

1.00 

1.75 

DCMC 

1.00 

3.00 

2.22 

2.00 

.54 

2.00 

2.00 

Prepare 
Estimates 

1.00 

4.00 

1.94 

2.00 

.77 

1.00 

2.00 

USAF 

1.00 

3.00 

1.88 

2.00 

.62 

1.25 

2.00 

Source:  Develop 

ed  by  : 

researc 

her . 

151 


2.    Specifying  Methods,  Models,  or  Software 

Question  6b,  shown  below,  deals  with  more  aggressive  use 

of  parametric  cost  estimating  methods  by  requiring  them  to  be 

used  when  the  buying  organization  feels  that  they  would  be 

helpful. 

In  solicitations,  buying  organizations  should,  specify 
parametric  cost  estimating  methods,  models,  or  software 
for  cost  proposal  construction. 

The  average  response  to  this  statement  was  to  disagree.   An 

average  summary  value  was  2.73  and  the  median  value  of  3 

indicate  this  sentiment.   The  three  groups  whose  data  are 

shown  in  Table  54  all  disagree  with  this  statement.   All  three 

groups'  responses  are  noteworthy;  the  control  group  because  of 

their  long  study  of  the  appropriate  uses  of  parametric  cost 

estimating  methods  and  both  contracting  officer  groups  because 

of  their  experience  in  generating  solicitations. 

Regression  analyses  of  the  GS/SES  level  and  business  size 

responses  show  a  relationship  exists.    The  equations  and 

statistics 

GS/SES  Level 

Y  =  .286(X)  -  1.45         R2  =  .10 

(.14)     (1.90) 

Business  Size 

Y  =  .471 (X)  -  3.71         R2  =  .03 

(.20)     (2.72) 

demonstrate  positive  relationships  between  responses  and 
increasing  Government  rank  and  also  increasing  business  size. 
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The  more  senior  the  respondent  the  more  they  disagreed  with 

the  statement  and  the  larger  the  business  the  more  that  they 

disagreed  with  the  statement.   It  is  very  clear  that  neither 

the  Government  nor  industry  wants  the  Government  to  specify 

cost   estimating  methods   to   be   used  in   response   to   a 

solicitation. 

Table  54 
Question  6b  Group  Data 


Min 

Max 

M 

Median 

a 

Ql 

Q3 

Control 
Group 

2.00 

4.00 

3.18 

3.00 

.60 

3.00 

4.00 

USN 

Contracting 

Officers 

2.00 

3.00 

2.75 

3.00 

.45 

2.25 

3.00 

USAF 

Contracting 

Officers 

3.00 

4.00 

3.17 

3.00 

.41 

3.00 

3.25 

Source:  Developed  by  researcher. 

3.    Offeror  Discretion  in  Using  Parametrics 

Question  6c  records  participant  opinions  about  the 

statement  that  follows: 

In  solicitations,  buying  activities  should  leave 
parametric  cost  estimating  methods,  models,  or  software 
selection  to  contractor  discretion. 

Leaving  the  use  of  cost  estimating  methods  to  be  used 
completely  up  to  the  seller's  discretion  was  agreed  to  by  111 
of  the  respondents.  This  produced  a  median  value  of  2  and  an 
average  response  value  of  2.33.   The  analysis  of  variance  with 
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the  DoD  component  data  showed  a  relationship  was  present  in 
the  information.  Of  the  DoD  job  classification  groups  only 
the  NASA  and  USAF  groups,  22  participants,  clearly  agreed  with 
this  statement.  The  data  to  support  this  are  shown  in  Tables 
55  and  56. 

With  the  exception  of  the  NASA  and  USAF  groups,  all  of 
the  other  job  classification  groups  were  split  between 
agreement  and  disagreement.  The  DCMC  response  data  in  Table 
55  is  representative  of  these  groups.  Unexpectedly,  there  was 
also  no  consensus  among  the  industry  participants  regarding 
this  statement  as  there  was  in  statement  6a. 


Table  55 
Question  6c  Group  Data 


Min 

Max 

M 

Median 

o 

Ql 

Q3 

DCMC 

1.00 

3.00 

2.24 

2.00 

.60 

2.00 

3.00 

NASA 

1.00 

2.00 

1.80 

2.00 

.48 

1.50 

2.00 

USAF 

1.00 

3.00 

1.88 

2.00 

.70 

1.00 

2.00 

Source: 

Develops 

id  by  r 

esearc 

tier. 

Table  56 
Analysis  of  Variance  For  Question  6c  Data 


DoD  Component 

DF 

Ed2 

G 

F 

P 

Between  Groups 

9 

9.77 

1.63 

2.57 

.02 

Within  Groups 

144 

91.20 

.63 

Source:  Developed  by  researcher. 
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4 .    Required  Cost  Breakdown  Formats 

Respondents  to  question  6d  were  asked  to  provide  their 

opinions  about  how  parametrics  might  change  data  presentation 

for  solicitations  through  the  following  statement: 

In  solicitations,  buying  activities  should  require  a 
specific  cost  breakdown  format  for  parametric  cost 
estimate  based  proposals. 

This  statement  addresses  the  issues  in  the  question: 

If  parametric  cost  based  estimates  are  used  in  a 
proposal,  should  the  buyer  state  the  format  that  the  cost 
information  is  to  be  presented? 

With  so  many  different  systems  producing  estimates,  some 
standardization  may  be  helpful  in  evaluating  proposals  in  a 
timely  manner.  A  specified  cost  breakdown  format  is  one 
method  of  standardization  that  is  available. 

The  majority  (108)  of  the  respondents  agreed  to  this 
premise  producing  an  average  response  value  of  2.35  and  median 
response  of  2.  Analysis  of  variance  on  the  job  classification 
responses  showed  that  a  relationship  between  the  responses  and 
the  classifications  was  present.  Tables  57  and  58  contain  the 
data  pertinent  to  the  question  6d  statement.  The  auditor, 
program  staff,  program  manager,  and  contract  negotiator 
groups,  54  respondents  total,  all  indicated  agreement  with 
this  statement.  These  groups  might  all  be  part  of  a  source 
selection  team  and  see  a  value  in  having  a  specified  format. 
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Another  significant  group  in  the  source  selection  process,  the 
contracting  officers,  did  not  come  to  a  consensus  on  either 
agreement  or  disagreement  with  the  statement,  however. 


Table  57 
Question  6d  Group  Data 


Min 

Max 

V- 

Media 
n 

a 

Ql 

Q3 

Auditors 

1.00 

3.00 

2.06 

2.00 

.65 

2.00 

2.25 

Negotiators 

1.00 

2.00 

1.67 

2.00 

.52 

1.00 

2.00 

DCAA 

1.00 

4.00 

2.06 

2.00 

.63 

2.00 

3.00 

Contracting 
Officers 

1.00 

4.00 

2.50 

2.00 

.73 

2.00 

3.00 

Program  Staff 

1.00 

3.00 

2.00 

2.00 

.71 

1.50 

2.00 

Program 
Managers 

1.00 

3.00 

1.86 

2.00 

.69  ' 

1.00 

2.00 

Source:  Developed  by  researcher. 


Table  58 
Analysis  of  Variance  For  Question  6d  Data 


Job  Description 

DF 

Zd2 

O2 

F 

P 

Between  Groups 

9 

11.19 

1.24 

2.05 

.04 

Within  Groups 

159 

96.59 

.61 

Source:  Developed  by  researcher. 

Regression  analysis  with  the  GS/SES  level  and  business 
size  group  data  showed  positive  relationships  in  the  data. 
The  equations 


GS/SES  Level 
Y  =  .300(X)  -  1.91 
(.14)     (2.01) 


R2  =  .10 


156 


Business  Size 

Y  =  .484  (X)  -  4.61         R2  =  .25 
(.18)     (2.45) 

describe  these  relationships.    Again,   as  one's  rank  or 

business  size  increases  so  does  the  response  value  to  the 

question.   These  results  indicate  that  experience  with  the 

methods  would  lead  to  allowing  the  seller  discretion  in  not 

only  their  use  but  in  how  they  choose  to  present  the  data  from 

those  estimates. 

5.    Cost  and  Technical  Proposal  Relationships 

A  second  area  of  concern  in  evaluating  proposals  is  that 

of  having  the  information  in  each  section  correspond  to  one 

another.   The  following  statement  addresses  this  issue: 

In  solicitations,  buying  activities  should  require  a 
specified  format  to  extract  labor  hours,  subcontractor, 
or  material  quantity  estimates  for  technical  proposal 
evaluation. 

A  standard  format  required  by  the  buyer  stating  that  format 
could  provide  that  concurrency.  Similar  results  to  the  6d 
statement  were  observed.  An  average  response  value  of  2.37 
and  a  median  value  indicate  agreement  with  the  statement  by 
the  entire  group  of  participants.  The  group  of  contract 
negotiators  had  a  consensus  that  they  strongly  agreed  with 
this  statement.  Their  group  response  was  the  only  group 
response  to  reach  any  type  of  consensus  of  all  of  the 
demographic  categories  identified. 
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6.    Statistical  Evaluation  Criteria 

Determining  what  criteria  will  be  used  to  evaluate  the 

validity  of  the  CERs  used  to  produce  estimates  included  in  a 

proposal  and  stating  them  in  the  solicitation  is  another 

method  of  creating  a  level  playing  field.   Statement  6f,  as 

shown  below,  examines  this  area: 

In  solicitations,  buying  activities  should  state  criteria 
used  to  evaluate  the  statistical  significance  of  cost 
estimating  relationships  used  to  support  a  parametric 
cost  estimating  model. 

A  great  deal  of  agreement  was  noted  for  this  statement.   The 

average  response  value  was  2.06  and  the  median  value  was  a  2, 

largely  a  result  of  137  respondents  electing  to. either  agree 

or  strongly  agree  with  the  statement.   The  group  data  in  Table 

59  show  a  wide  range  of  acguisition  activities  across  the 

entire  acquisition  community  who  agree  with  the  statement. 

Table  59 
Question  6f  Group  Data 


Min 

Max 

M 

Median 

a 

Ql 

Q3 

Program  Staff 

1.00 

2.00 

1.50 

1.50 

.58 

1.00 

2.00 

Cost 
Estimators 

1.00 

3.00 

1.97 

2.00 

.55 

2.00 

2.00 

Prepare  For 
Negotiations 

1.00 

4.00 

2.00 

2.00 

.74 

2.00 

2.00 

Prepare 
Estimates 

1.00 

3.00 

2.03 

2.00 

.65 

2.00 

2.00 

USAF 

1.00 

4.00 

2.24 

2.00 

.66 

2.00 

2.25 

Source:  Develope 

;d  by  r 

esearc 

her. 
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Information  regarding  CER  validity  could  be  interpreted  to  be 

required  by  FAR  part  15.406-5  in  that  it  could  be  defined  as 

a  significant  factor  that  might  be  considered  in  awarding  a 

contract . 

7.    Traditional  or  Parametrics  Methods  in  Solicitations 

Statement  6g,  show  below,  asks  survey  participants  their 

opinion  about  the  concept  of  fairness  in  a  solicitation  if 

parametric  methods  are  either  specified  or  excluded. 

In  solicitations,  buying  activities  should  require  cost 
proposals  based  on  either  parametric  cost  estimating 
methods  or  traditional  methods. 

A  source  selection  plan  should  not  favor  one  proposal  over 
another  simply  by  the  way  the  proposal  evaluation  plan  is 
constructed.  The  plan  must  make  sure  procedures  do  not  act 
to  exclude  or  discourage  potential  offerors  from  submitting 
proposals.  The  statement  addresses  the  problem  of  fairly 
evaluating  a  parametric-based  cost  proposal  with  one  produced 
using  more  traditional  detailed  methods.  Can  an  evaluation 
plan  be  constructed  that  will  fairly  evaluate  both  types  of 
cost  estimates  or  must  the  buyer  specify  which  an  offeror 
should  use  because  they  cannot  be  evaluated  fairly? 

There  was  no  agreement  on  this  question  at  the  level  of 
the  entire  pool  of  participants  nor  on  an  individual  group 
level.  The  aggregate  response  average  value  was  2.55  and  a 
median  value  of  3  was  observed.   These  data  indicate  that  the 
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group  disagreed  in  general  with  the  statement  but  that  they 
were  effectively  split  over  agreeing  and  disagreeing.   There 
were  also  a  significant  number  (51)  of  no  opinion  or  no 
responses  to  this  question. 
D .    SUMMARY 

The  results  from  the  analysis  of  the  statements  in 
questions  five  and  six  provide  a  number  of  areas  that  could 
be  affected  by  the  use  of  parametric  cost  estimating  methods. 
Chief  among  these  areas  is  the  contractor  cost  estimating 
system.  The  systems  will  respond  to  the  encouragement  that 
the  Federal  Government  members  of  the  DoD  acquisition 
community  provides .  There  was  universal  agreement  that  buying 
activities  should  encourage  the  use  of  parametric  cost 
estimating  methods  but  not  require  them. 

One  of  the  most  effective  methods  of  encouraging  the 
use  of  the  methods  is  in  the  source  selection  process.  An 
evaluation  of  a  parametric-based  proposal  that  is  viewed  by 
the  offeror  to  be  unfair  could  be  taken  as  discouragement. 
The  trend  should  remain  with  telling  the  offerors  what  is 
required  and  letting  them  propose  methods  of  achieving  that 
goal.  In  that  regard,  source  selection  plans  should  be 
constructed  to  allow  any  reasonable  type  of  cost  proposal 
submitted  to  be  fairly  evaluated. 
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Required  formats  for  data  submission  in  terms  of  the 
cost  proposal  or  technical  proposal  were  viewed  by  the  group 
as  useful  in  creating  a  level  playing  field.  These  actions 
have  the  advantage  of  letting  the  offerors  meet  the  goal  that 
is  required  rather  than  telling  them  how  it  is  to  be 
accomplished.  They  prevent  the  Government  from  dictating 
specific  methods  to  be  used  while  every  offeror  has  some  idea 
of  how  the  information  provided  will  be  used.  They  have  the 
added  benefit  of  providing  some  consistency  throughout  the 
proposals  which  speeds  up  the  evaluation  process. 

If  parametric  estimating  methods  are  acceptable  for 
use  as  a  basis  for  a  cost  proposal,  the  proposal  must  state 
how  they  will  be  evaluated  when  compared  with  other  methods. 
FAR  part  15.605(c)  (d)  (1)  addresses  the  issue  of  evaluation 
factors  under  which  this  issue  falls. 


...  at  a  minimum,  the  solicitation  shall  clearly 
state  the  significant  evaluation  factors,  such  as 
cost  or  price,  cost  or  price-related  factors,  past 
performance  and  other  non-cost  or  non-price-related 
factors,  and  any  significant  subfactors,  that  will 
be  considered  in  making  the  source  selection.  [Ref 
13:part  15.605] 


A  buying  organization  must  consider  constructing  their 
solicitation  to  include  the  possibility  of  parametric 
estimate-based  cost  proposals  if  all  proposals  are  to  be 
evaluated  fairly.    The  members  of  the  source  selection 
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evaluation  board  should  also  consider  this  issue  as  cost  is 
always  an  evaluation  factor. 
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VIII.  PROPOSAL  ANALYSIS  AND  NEGOTIATIONS 

A.  PURPOSE 

Chapter  VIII  is  an  analysis  of  the  responses  to  questions 
seven  through  ten.  These  survey  questions  and  the  last  four 
interview  questions  address  issues  that  may  arise  during  the 
evaluation  of  a  parametric-based  proposal  and  subsequently  the 
negotiations  that  may  result.  The  four  survey  question 
responses  were  graded  along  the  same  five  point  scale  used  in 
questions  five  and  six.  With  the  exception  of  question  nine, 
all  of  the  questions  were  a  series  of  statements  upon  which 
participants  recorded  their  opinions. 

B.  COST  PROPOSAL  SUPPORTING  DATA 

Question  seven  is  a  three  part  series  of  statements  that 

again  address  the  first  subsidiary  research  question. 

What  changes  could  offerors  make  to  their  proposal 
preparation,  submission,  and  support  processes  to 
accommodate  using  parametric  methods,  techniques  or 
software  in  estimating  potential  contract  costs? 

In  the  three  statements  for  question  seven,  an  offeror  has 

decided  to  provide  a  parametric-based  cost  estimate  in 

response  to  a  solicitation.    The  question  addresses  the 

information  that  might  be  submitted  to  substantiate  that 

estimate.   The  Government  goal  in  obtaining  information  about 

the  estimate  centers  around  regenerating  that  estimate  to 

understand  how   it   was   generated,   its   limitations,   and 
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strengths.  Providing  data  to  do  this  is  a  vital  part  of 
submitting  a  successful  proposal. 

1 .    Software  and  Input  Data 

Statement  7a  addresses  one  of  the  ways  that  an  offeror 

could  chose  or  be  required  to  provide  supporting  information 

for  a  cost  proposal. 

A  potential  contractor  should  provide  the  parametric  cost 
estimating  model  software  and  input  data  used  to  support 
a  cost  proposal. 

Providing  the  software  and  input  data  used  to  produce  an 

estimate  is  one  way  to  meet  the  Government  goal  of  being  able 

to  reproduce  the  estimate.   Without  these  tools  it  cannot  be 

accomplished.   There  was  a  strong  agreement  to  this  statement 

in  the  responses.   The  response  average  was  1.81,  generated  in 

large  part  by  the  large  number  of  people  who  answered  strongly 

agree  (66)  and  agree  (95)  to  this  statement.   Responses  to 

this  statement  also  had  a  small  standard  deviation  (.76) 

indicating   a   consensus   among   the   respondents   on   this 

statement.   The  group  data  in  Table  60  provides  a  sample  of 

the  groups  that  agreed  to  this  statement  within  the  DoD 

acquisition  community. 

Agreement  to  this  statement  covered  the  entire 

spectrum  of  functional  areas  that  might  be  tasked  to  evaluate 

a  cost  proposal.   All  responded  that  this  information  should 

be  provided  by  an  offeror  in  support  of  a  cost  proposal.   The 
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industry  response  to  the  statement  was  not  as  clear.   There 

was  no  consensus  in  the  industry  response  as  a  whole  nor  in 

the  group  that  set  corporate  policy. 

Table  60 
Question  7a  Group  Data 


Min 

Max 

M 

Median 

o 

Ql 

Q3 

Auditors 

1.00 

4.00 

1.65 

2.00 

.70 

1.00 

2.00 

Cost 
Analysts 

1.00 

4.00 

1.73 

2.00 

.72 

1.00 

2.00 

Contracting 
Officers 

1.00 

3.00 

1.83 

2.00 

.54 

1.50 

2.00 

Cost 
Estimators 

1.00 

3.00 

1.83 

2.00 

.72 

1.00 

2.00 

Engineers 

1.00 

3.00 

1.91 

2.00 

.70 

1.00 

2.00 

Source:  Developed  by  researcher. 

2.    Justifying  Parametric  Use 

Beyond  providing  the  software  and  input  data,  there 

should  be  a  supportable  reason  to  use  parametric  cost 

estimating  methods.    They  should  be  applicable  to  the 

procurement  action  in  question  and  the  work  relatable  to  the 

databases  used.    Shown  below,  statement  7b  examines  this 

issue. 

A  potential  contractor  should  provide  justification  to 
support  the  use  of  parametric  cost  estimating  models  and 
input  data  in  his  cost  proposal. 

The  majority  of  the  participants  answered  that  they  agreed 

(107)  or  strongly  agreed  (64)  with  the  statement.   The  average 

response  value  was  1.75,  a  standard  deviation  of  .65  was  noted 
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and  agree  was  the  median  response.  The  widespread  agreement 
to  this  statement  includes  almost  every  functional  group  in 
the  DoD  acquisition  community.  The  major  functional  areas  of 
the  community  that  typically  are  involved  in  evaluating 
proposals  are  shown  from  buying  organizations  to  the  industry. 
All  of  the  groups  shown  in  Table  61  have  a  consensus  on  the 
issue. 


Table  61 
Question  7b  Group  Data 


Min 

Max 

M 

Median 

a 

Ql 

Q3 

Control  Group 

1.00 

3.00 

1.73 

2.00 

.65 

1.00 

2.00 

Cost  Analysts 

1.00 

3.00 

1.73 

2.00 

.60 

1.00 

2.00 

Contracting 
Officers 

1.00 

3.00 

1.76 

2.00 

.54 

1.00 

2.00 

Cost 
Estimators 

1.00 

2.00 

1.67 

2.00 

.49 

1.00 

2.00 

Engineers 

1.00 

3.00 

2.00 

2.00 

.74 

1.25 

2.75 

DCAA 

1.00 

4.00 

1.71 

2.00 

.64 

1.00 

2.00 

DCMC 

1.00 

3.00 

1.86 

2.00 

.64 

1.00 

2.00 

Industry 

1.00 

4.00 

1.70 

2.00 

.82 

1.00 

2.00 

Source:  Develop 

ed  by  r 

esearc 

ier. 

3.    Data  and  Rationale 

Additional  information  that  is  critical  to  understanding 
a  parametric  cost  based  estimate  is  knowing  how  it  has  been 
validated  for  use.  Statement  7c  is  presented  below  followed 
by  the  analysis  of  the  data  collected  regarding  this  issue. 
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A  potential  contractor  should  provide  data  and  rationale 
used  to  validate  a  parametric  cost  estimating  model  for 
a  specific  set  of  circumstances  in  support  of  a  cost 
proposal . 

The  statement  indicates  the  offeror  should  provide  information 

about  how  the  model  was  modified,  the  database  updated,  and 

how  often  these  tasks  were  performed  to  give  the  buyers  the 

level  of  understanding  that  they  need.   The  groups  shown  in 

Table  62  again  show  the  level  of  agreement  with  this  statement 

across  the  DoD  acquisition  infrastructure. 


Table  62 
Question  7c  Group  Data 


Min 

Max 

V- 

Median 

a 

Ql 

Q3 

Program  Staff 

2.00 

2.00 

2.00 

2.00 

.00 

2.00 

2.00 

Control  Group 

1.00 

3.00 

1.73 

2.00 

.65 

1.00 

2.00 

Cost  Analysts 

1.00 

3.00 

1.66 

2.00 

.52 

1.00 

2.00 

Contracting 
Officers 

1.00 

3.00 

1.68 

2.00 

.52 

1.00 

2.00 

Cost 
Estimators 

1.00 

2.00 

1.67 

2.00 

.49 

1.00 

2.00 

Engineers 

1.00 

3.00 

1.75 

2.00 

.62 

1.00 

2.00 

DCAA 

1.00 

4.00 

1.57 

2.00 

.63 

1.00 

2.00 

DCMC 

1.00 

3.00 

1.79 

2.00 

.63 

2.00 

3.00 

Industry 

1.00 

4.00 

1.65 

2.00 

.75 

1.00 

2.00 

NASA 

1.00 

3.00 

1.75 

2.00 

.54 

1.00 

2.00 

U.S.  Navy 

1.00 

2.00 

1.67 

2.00 

.48 

1.00 

2.00 

USAF 

1.00 

2.00 

1.71 

2.00 

.47 

1.00 

2.00 

Source:  Develops 

;d  by  r 

esearc 

tier. 
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Agreement  to  this  statement  was  also  strong  and  widely 
represented  in  the  DoD  community.  Strongly  agree  was  selected 
by  71  and  agree  by  106  respondents  leading  to  an  average 
response  value  of  1.67,  a  small  standard  deviation  of  .61  and 
a  median  value  of  2. 

Offerors  should  provide  information  described  in  the 
three  statements  in  question  seven.  Gathering  these  data 
could  require  new  information  systems  to  be  used  in  industry 
to  capture,  collate,  and  display  this  information.  The 
responses  show  support  from  industry  to  provide  this 
information  as  well  as  a  desire  by  those  charged  with 
evaluating  proposals  to  have  these  data  available. 
C.    DCAA  SUPPORT 

After  a  proposal  is  submitted,  it  is  then  analyzed  in 

accordance  with  the  evaluation  plan.   In  a  four  part  series  of 

statements,  question  eight  addresses  possible  functions  that 

may  need  to  be  performed  and  who  is  best  suited  to  do  them. 

These   statements   address   issues   raised   in   the   second 

subsidiary  research  question. 

What  impacts  could  emphasis  on  using  cost  estimates  based 
on  parametric  estimating  techniques  have  on  source 
selection  criteria  and  on  the  evaluation  of  proposals  by 
the  DoD? 

Each  statement  assumes  that  DCAA  is  the  best  organization  to 
perform  the  task  identified.   The  respondents  are  asked  to 
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fill  in  another  organization  if  they  feel  that  DCAA  is  not  the 
appropriate  action  agency  for  the  function  in  question. 
Information  about  the  written  responses  will  be  provided  with 
the  analysis  for  each  question. 

1.    Establishing  Validity  Metrics 

Determining  the  validity  of  a  parametric  model  to  a  given 

set  of  circumstances  is  critical  to  understanding  the  outputs 

of  that  model,  their  uses,  and  limitations.   Statement  8a, 

shown  below,  addresses  this  concern. 

DCAA  should  establish  metrics  to  determine  a  parametric 
cost  estimating  model's  validity  in  a  given  situation. 

Defining  measurement  criteria  to  gauge  the  effectiveness  of 

a  model  in  a  particular  situation  refers  to  the  concept  of 

calibration   defined   in   the   Parametric   Cost   Estimating 

Handbook.   If  models  are  modified  for  use,  descriptions  of  how 

they  were  changed  to  reflect  the  situation  in  which  they  will 

be  used  should  be  provided  to  proposal  evaluators.    This 

statement  asks  the  respondent  to  determine  who  should  set 

forth  these  criteria. 

There  was  a  large  amount  of  agreement  with  this  statement 

indicating  that  DCAA  should  perform  this  function.    The 

average  response  value  of  2.25  results  from  the  large  number 

of  participants   (121)   agreeing  with  the  statement.    The 

program  staff,  U.S.  Army  cost  analyst,  and  USAF  contracting 

officer   groups   represent   agreement   within   the   survey 
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participants.  This  is  contrasted  with  the  disagreement  that 
the  control  group  expressed.  The  statement  group  data  are 
shown  in  Table  63. 


Table  63 
Question  8a  Group  Data 


Min 

Max 

¥■ 

Median 

a 

Ql 

Q3 

Program  Staff 

2.00 

2.00 

2.00 

2.00 

.00 

2.00 

2.00 

Control  Group 

2.00 

4.00 

2.73 

3.00 

.77 

2.00 

3.00 

U.S.  Army 
Cost  Analysts 

1.00 

3.00 

1.93 

2.00 

.46 

2.00 

2.00 

USAF 

Contracting 

Officers 

1.00 

2.00 

1.83 

2.00 

.41 

1.75 

2.00 

Source:  Develope 

:d  by  r 

esearc 

her. 

Regression  analysis  also  showed  a  disagreement  in  the 
responses.   The  business  size  regression  equation 


Business  Size 
Y  =  .733  (X)  -  7.64 
(.17)    (2.36) 


R2  =  .51 


indicates  that  as  the  business  size  increases  so  does  the 
disagreement  with  the  statement.  The  same  analysis  performed 
with  the  GS/SES  level  data  shows  an  opposite  relationship. 
The  equation  and  data 


GS/SES  Level 
Y  =  -.418  (X)  +  8.28 
(.13)    (1.84) 


R2  =  .23 


show  a  negative  relationship.   As  the  Government  rank  rises  so 
does  the  level  of  agreement  with  the  statement.   The  division 
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is   clearly   along   a   Government- Industry   split   in   the 
acquisition  community. 

There  are  two  areas  that  may  be  the  source  of  the 
disagreement.  The  respondents  may  feel  that  the  function  does 
not  need  to  be  performed  or  that  DCAA  is  not  the  appropriate 
organization  to  do  the  work.  Since  the  functions  described 
appear  as  a  central  concept  in  the  Parametric  Cost  Estimating 
Handbook  and  several  papers  written  by  members  of  the  control 
group,  the  likely  source  of  the  disagreement  is  in  the  area  of 
who  should  do  the  work.  Two  answers  were  written  in  by  the 
control  group.   Three  people  recommended  that  either 

1.  DCMC  should  perform  this  function,  (2)  or  that 

2.  This  work  should  be  left  to  the  contractor's 
discretion.  (1) 

These  two  responses  were  also  prominent  in  the  aggregate  of 
written  responses  to  this  statement. 

In  all  of  the  surveys  returned,  six  groups  other  than 
DCAA  were  identified  to  perform  this  function: 

1.  Buying  Organizations  (14) 

2.  Offerors  (12) 

3.  DCMC  (15) 

4.  Cost  Analysis  Improvement  Group  (CAIG)  (2) 
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5.  A  DCMC,    DCAA,    Buying  Organization,    and  Offeror  Team 
(2) 

6.  Program  Managers    (1) 


The  majority  of  the  added  recommendations  concentrated  on 
DCMC,  the  buying  organization,  or  the  contractor  performing 
the  work.  DCAA  and  the  control  group  recommend  leaving  this 
to  the  contractor' s  discretion  rather  than  assigning  it  as  a 
DCAA  responsibility.  DCMC  and  the  buying  organizations,  USAF, 
U.S.  Army,  and  U.S.  Navy,  would  prefer  to  take  on  this 
function  rather  than  task  DCAA  with  it. 

Both  possible  issues  of  contention  in  this  statement  are 
addressed  by  Dennedy,  Deutsch,  and  Hertling  in  their  article. 
They  state  that  this  function  must  be  accomplished  and 
recommend  that  it  be  done  prior  to  using  the  models  to 
generate  estimates.  [Ref  8:p.l]  They  also  directly  address 
the  topic  of  who  should  perform  this  verification  activity  for 
models  generated  by  the  potential  offeror. 


The  parametric  model  (or  CER  model)  should  be 
developed  at  the  local  level  in  coordination  with 
the  cognizant  ACO  and  DCAA  auditors.  Ideally,  a 
team  made  up  of  ACO  pricing,  buying  office 
representatives  and  the  contractor' s  parametric 
model  builders  should  develop  the  model  jointly  in 
an  integrated  product  team  (IPT)  process.  The 
joint  team  should  review  and  validate  the  database 
upon  which  the  model  is  based,  the  logic  and 
statistical  validity  of  the  model  itself  and  how 
the  model  will  fit  into  the  contractor' s  estimating 
system.    DCAA,  as  an  independent  examiner,  will 
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provide   input   on   the   contractor's   controls, 
database,  and  updating  procedures.  [Ref  8:  p.l] 


In  this  way,  the  teaming  concept  and  IPTs  are  introduced  at  a 
very  early  point  in  the  procurement  process.  The  write-in 
responses  indicating  that  a  team  would  be  beneficial  came  from 
both  DCAA  and  DCMC  for  this  segment  of  question  eight.  While 
teaming  may  be  the  best  solution  to  the  problem  of  who  should 
validate  and  how  a  model  should  be  validated,  the  buying 
organizations  do  not  seem  to  have  considered  that  idea  at  this 
point  in  responding  to  this  statement. 

2 .    Updating  Models 

A  second  facet  of  model  validation  concerns  when  the 

model  is  modified  or  database  updated  and  who  should  make  this 

decision.    Survey  participants  were  asked  to  record  their 

opinions  to  statement  8b: 

DCAA  should  establish  criteria  to  determine  when  a 
parametric  cost  estimating  model  should  be  updated. 

The  same  basic  issues  that  are  evident  in  the  concept  of 

calibration  apply  to  when  a  model  should  be  calibrated.   The 

participants  agreed  to  this  statement  in  the  majority;  126 

people  either  strongly  agreed  or  agreed  with  the  statement. 

An  average  response  value  of  2.20  and  median  response  of  2  was 

observed.   The  data  shown  in  Table  64  represent  the  agreement 

throughout  the  acquisition  community  on  this  issue.    The 

unanimous  agreement  in  the  program  staff  group  and  degree  of 
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consensus  reached  on  this  issue  in  the  other  groups  is 
noteworthy.  Industry  and  Government  members  of  the 
acquisition  community  agree  that  someone  should  set  criteria 
to  determine  when  models  should  be  updated.  A  significant 
portion  (20%)  of  the  respondents  recommended  that  someone 
other  than  DCAA  be  tasked  with  this  function. 


Table  64 
Question  8b  Group  Data 


Min 

Max 

M 

Median 

a 

Ql 

Q3 

Program 
Staff 

2.00 

2.00 

2.00 

2.00 

.00 

2.00 

2.00 

Control 
Group 

1.00 

3.00 

2.00 

2.00 

.63 

2.00 

2.00 

U.S.  Army 

Cost 

Analysts 

1.00 

3.00 

2.00 

2.00 

.52 

2.00 

2.00 

Industry 

Cost 

Analysts 

1.00 

2.00 

1.86 

2.00 

.38 

2.00 

2.00 

Contracting 
Officers 

1.00 

3.00 

2.00 

2.00 

.61 

2.00 

2.00 

Source:  Developed  by  researcher. 

With  the  exception  of  the  program  managers,  the  list  of 
other  recommendations  is  the  same  as  in  the  previous 
statement.  The  buyer  (9),  offeror  (5),  DCMC  (18),  and  a  team 
arrangement  (4)  were  the  groups  most  often  recommended  to 
perform  this  work.  Of  the  responses  from  buying 
organizations,  the  write-in  responses  recommended  either  that 
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they  should  determine  when  a  proposed  model  requires  updating 
or  that  DCMC  take  on  that  work.  The  DCAA  written 
recommendations  indicate  that  the  offeror  should  perform  this 
work. 

The  Dennedy  et  al  article  does  not  specifically  address 
the  issue  of  when  a  model  should  be  updated  but  the  concept  is 
included  in  the  duties  listed  for  the  team  members.  Inherent 
in  the  article  is  an  assumption  that  the  offeror  has  taken  the 
initiative  to  keep  the  model  current  and  that  it  is  applicable 
to  the  given  situation.  The  Government  functions  in  a  review 
and  evaluation  mode  with  three  organizations,  DCAA,  DCMC,  and 
the  buying  office,  performing  various  review  .duties.  The 
currency  of  the  model  is  considered  in  the: 


1 .  DCAA  review  of  calibration  and  validation  of  the 
tools, 

2.  DCMC  technical  review  of  the  contractor's  cost 
modeling  technique,  and 

3.  Buying  office  review  of  the  offeror's  assumptions 
made  in  the  model.  [Ref  8: p. 3] 


Ideally,  this  would  be  performed  in  the  team  environment 
presented  in  the  previous  statement  analysis.  However,  the 
written  responses  to  this  question  indicate  that  the  buying 
organizations,  DCMC  and  DCAA,  do  not  view  teaming  as  an  answer 
to  this  problem. 
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3 .    Model  Validation 

The  issues  addressed  in  statement  8c,  shown  below,  are 

centered  around  the  concept  of  model  validation. 

DCAA  should  verify  that  a  parametric  cost  estimating 
model  accurately  predicts  costs  for  the  expected  contract 
work. 

While  there  is  no  way  to  be  completely  certain,  until  after 
the  work  is  done,  that  a  model  has  accurately  predicted  the 
costs  for  a  project,  there  are  methods  that  can  be  used  to 
determine  its  accuracy.  This  process  is  described  in  detail 
as  validation  in  the  Parametric  Cost  Estimating  Handbook.  It 
may  include: 

1.  Calibration  of  the  model  to  historical  cost  data, 

2.  Estimating  the  cost  of  past  completed  projects,  and 

3.  Comparing  these  estimates  with  actual  costs  to 
demonstrate  acceptable  accuracies,  [Ref  37:p.l8] 

as  steps  in  a  validation  process.  The  average  response  value 
of  1.93  and  median  value  of  2  show  a  strong  level  of  agreement 
with  the  survey  statement.  The  strong  support  for  the 
statement  is  shown  throughout  the  survey  respondents  in  the 
group  data  presented  in  Table  65.  Program  personnel, 
contracting  officers,  and  industry  groups  all  agreed  that  DCAA 
should  perform  this  verification.  That  a  smaller  number  of 
written  recommendations  was  received  for  this  statement 
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indicates  agreement  that  DCAA  is  also  the  organization  best 

suited  for  doing  this  work. 

Table  65 
Question  8c  Group  Data 


Min 

Max 

M 

Median 

a 

Ql 

Q3 

Program 
Staff 

2.00 

2.00 

2.00 

2.00 

.00 

2.00 

2.00 

U.S.  Army 

Cost 

Analysts 

1.00 

4.00 

1.94 

2.00 

.68 

2.00 

2.00 

Industry 

Cost 

Analysts 

1.00 

3.00 

1.86 

2.00 

.69 

1.00 

1.00 

Contracting 
Officers 

1.00 

3.00 

1.83 

2.00 

.59 

1.00 

2.00 

Program 
Managers 

1.00 

2.00 

1.83 

2.00 

.41  - 

1.75 

2.00 

Corporate 
Policy 

1.00 

2.00 

1.75 

2.00 

.50 

1.25 

2.00 

Prepare  For 
Negotiations 

1.00 

4.00 

1.83 

2.00 

.67 

1.00 

2.00 

Source:  Developed  by  researcher. 

Dennedy,  Deutsch,  and  Hertling  also  state  clearly 
the  function  should  be  performed  by  DCAA  but  they  emphasize 
that  ideally  it  would  be  done  in  a  team  environment.  [Ref 
8:p.3]  The  results  of  the  recommendations  received  show  that/ 
again,  buyers  would  prefer  to  retain  this  work  or  task  DCMC 
with  it.  DCMC  written  recommendations  are  that  they  would 
prefer  to  retain  it.  It  should  be  noted  that  only  15%  of  the 
respondents  chose  to  provide  a  written  response.   Of  the  total 
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number  of  written  recommendations  to  this  statement,  DCMC 
provided  six.  DCAA  is  well-suited  to  perform  this  task  in 
their  traditional  role,  and  the  lack  of  DCAA  written  responses 
shows  an  agreement  to  take  on  this  work. 

4 .    Model  Operation 

The  basis  of  analyzing  a  cost  proposal  is  understanding 

the  proposal  in  detail  and  the  relationships  within  it.   Using 

an  offeror's  model  to  generate  cost  estimates  is  one  method  of 

applying  the  information  provided  about  that  model  to  ensure 

this  level  of  understanding.   The  final  statement  of  question 

eight  asks  the  respondents  to  give  their  opinion  about  the 

issues  in  this  statement: 

DCAA  should  run  a  contractor' s  parametric  cost  estimating 
model  for  purposes  of  analyzing  the  cost  proposal. 

There  was  again  a  great  deal  of  agreement  with  DCAA  being 

tasked  with  operating  an  offeror's  model  for  these  purposes. 

There  were  129  responses  in  agreement  with  this  statement 

which  produced  an  average  response  value  of  2.08  with  a  .74 

standard  deviation  and  a  median  value  of  2.   Table  66  presents 

the  group  data  for  this  statement.   Notably,  the  control, 

program  manager,  and  program  staff  groups  were  unanimous  in 

their  agreement  with  the  idea  that  DCAA  should  provide  this 

service.   One  major  buying  organization,  contracting  officers 

and  three  job  description  groups  reached  a  consensus  that  this 

service  should  be  provided. 
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Table  66 
Question  8d  Group  Data 


Min 

Max 

M 

Median 

a 

Ql 

Q3 

Control 
Group 

2.00 

2.00 

2.00 

2.00 

.00 

2.00 

2.00 

Program 
Staff 

2.00 

2.00 

2.00 

2.00 

.00 

2.00 

2.00 

U.S.  Army 

Cost 

Analysts 

1.00 

3.00 

2.00 

2.00 

.56 

2.00 

2.00 

Contracting 
Officers 

1.00 

3.00 

1.95 

2.00 

.52 

2.00 

2.00 

Program 
Managers 

2.00 

2.00 

2.00 

2.00 

.00 

2.00 

2.00 

U.S.  Army 

1.00 

4.00 

2.03 

2.00 

.58 

2.00 

2.00 

Prepare  For 
Negotiations 

1.00 

4.00 

1.93 

2.00 

.63 

2.00 

2.00 

Prepare 
Estimates 

1.00 

4.00 

2.11 

2.00 

.64 

1.00 

2.00 

Source:  Developed  by  researcher. 

Written  responses  to  this  statement  show  that  buyers  and 

DCMC  would  prefer  to  perform  this  work  themselves.   The  lack 

of  DCAA  written  responses  would  indicate  that  they  agree  with 

their   organization   being   tasked   with   this   function. 

Regression  analysis  performed  on  the  experience  data  presents 

the  following  relationship: 

Y  =  -.192 (X)  +  7.48        R2  =  .04 
(.07)     (2.06) 

The  weak  negative  relationship  indicates  that  as  experience 

increases  so  does  the  agreement  with  this  statement.   Stronger 
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support  for  this  statement  is  coming  from  more  experienced 
groups  such  as  the  prepare  estimates  work  classification 
group.  Notably  there  was  a  lack  of  consensus  in  the  industry 
and  corporate  policy  groups  on  the  issue  of  DCAA  operating  a 
parametric  cost  estimating  model  for  analysis  purposes. 

The  Dennedy  article  provides  a  description  of  a  possible 
mechanism  for  accomplishing  an  agency  running  an  offeror's 
model.  The  article  espouses  having  the  offeror  make  the  model 
available  to  the  Government  rather  than  providing  a  copy  of 
the  model  to  the  Government  as  discussed  in  guestion  7a.  The 
agency  responsible  for  operating  the  model  in  the  contractor 
facility  is  not  addressed,  however.  The  chief-  advantage  to 
this  solution  is  the  model  remains  a  part  of  the  estimating 
system  that  generated  the  offer  in  the  proposal  rather  than  a 
separate  piece  to  be  inserted  into  a  negotiating  position. 

5.    Summary 

Agreement  was  observed  for  all  four  of  the  guestion  eight 
statements.  Minor  areas  of  disagreement  seem  to  be  related  to 
who  would  perform  the  work  or  function  described  rather  than 
if  the  work  should  be  performed  at  all.  At  most,  25%  of  the 
respondents  chose  to  offer  a  recommendation  for  some 
organization  other  than  DCAA  to  be  tasked  with  the  work 
described  in  any  one  of  the  statements.  Additionally,  less 
than  25%  of  the  respondents  disagreed  with  any  one  of  the 


180 


statements  in  question  eight.   Written  responses  to  these 

statements  indicate  a  preference  for  two  groups,   DCMC  and  the 

buying  organizations,  taking  on  this  function  if  DCAA  does  not 

do  it.   This  desire  to  be  involved  in  these  aspects  of  the 

model  evaluation  could  be  beneficial.   The  team  environment 

described  by  Dennedy,  Deutsch  and  Hertling  is  a  viable  vehicle 

for  combining  the  experience,  knowledge,  and  needs  of  these 

groups  to  provide  a  single  face  to  industry  in  another  area  of 

the  acquisition  process  to  address  common  goals. 

D.    TECHNICAL  PROPOSAL  EVALUATION  IMPACTS 

Participants  to  question  nine,  shown  below,  were  asked  to 

record  their  opinions  about  the  information  that  parametric 

methods  could  provide  in  technical  proposal  evaluation. 

Construction  of  cost  estimating  relationships  used  in 
parametric  cost  estimate  based  proposals  should  be  used 
as  a  factor  in  evaluating  a  contractor's  level  of 
understanding  of  the  technical  requirements  in  a 
solicitation. 

Subsidiary  question  two  and  three  issues  are  addressed  in  this 

statement.   Source  selection  criteria  and  evaluation  factors 

should  take  the  entire  proposal  into  account.    Technical 

understanding  and  capability  can  be  revealed  in  the  way  the 

offeror   intends   to   manage   the   effort,   the   way   the 

relationships  in  the  cost  estimate  are  related  as  well  as  in 

the  verbiage  in  the  technical  proposal  itself. 
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The  average  response  value  of  2.37  and  median  value  of  2 

indicate  a  fair  amount  of  agreement  with  this  statement. 

Table  67  shows  that  the  program  manager  and  NASA  groups  were 

strongest  in  their  agreement  with  the  statement.    Buying 

organizations,  U.S.  Navy,   support  groups  in  the  acquisition 

community,   and   contracting   officers,   particularly   USAF 

contracting  officers,  also  agreed  with  this  premise. 

Table  67 
Question  9  Group  Data 


Min 

Max 

M 

Median 

a 

Ql 

Q3 

Contracting 
Officer 

1.00 

4.00 

2.18 

2.00 

.68 

2.00 

2.00 

USAF 

Contracting 

Officers 

2.00 

2.00 

2.00 

2.00 

.00  • 

2.00 

2.00 

Cost 
Estimators 

2.00 

4.00 

2.28 

2.00 

.49 

2.00 

2.25 

NASA 

1.00 

2.00 

1.86 

2.00 

.38 

2.00 

2.00 

U.S.  Navy 

1.00 

3.00 

2.00 

2.00 

.56 

2.00 

2.00 

Program 
Managers 

1.00 

2.00 

1.67 

2.00 

.52 

1.00 

2.00 

Auditors 

1.00 

4.00 

2.13 

2.00 

.73 

2.00 

3.00 

Technical 
Support 

2.00 

4.00 

3.67 

4.00 

1.5 

2.00 

4.00 

Source:  Developed  by  researcher. 

Analysis  of  variance  of  job  description  groups  revealed 
that  program  managers,  contracting  officers,  and  auditors,  69 
respondents,  agreed  that  CER  construction  should  be  considered 
as  a  technical  evaluation  factor.   It  also  showed  that  the 
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technical  support  participants  strongly  disagreed  with  this 

statement  although  they  do  represent  a  very  small  group  within 

the  pool  of  respondents.   This  is  an  interesting  response  from 

those  who  would  be  likely  to  be  assigned  technical  proposal 

evaluation  duties.   Table  68  contain  the  job  classification 

analysis  of  variance  data. 

Table  68 
Analysis  of  Variance  For  Question  9  Data 


Job  Description 

DF 

2d2 

O2 

F 

P 

Between  Groups 

9 

14.51 

1.61 

2.51 

.01 

Within  Groups 

134 

86.03 

.64 

Source:  Developed  by  researcher. 

Weak  negative  relationships  between  responses  and  both 
GS/SES  level  and  experience  were  also  noted.   The  equations 


GS/SES  Level 

Y  =  -.336(X)  +  7.29 

(.15)  (2.07) 

Experience 

Y  =  -.179(X)  +  7.45 

(.09)  (2.44) 


R2  =  .13 


R2  =  .02 


indicate  that  as  GS/SES  level  or  experience  increases  the 
agreement  with  the  statement  also  increases  slightly.  This 
also  supports  the  agreement  levels  seen  in  the  auditor,  cost 
estimating,  and  NASA  groups.  These  groups  are  among  the  more 
experienced  groups  of  all  the  demographic  categories  in  the 
respondent  pool. 
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The  agreement  with  question  nine  indicates  that 
consideration  should  be  given  to  this  aspect  of  technical 
proposal  evaluation.  With  the  strong  disagreement  from  the 
technical  support  community,  it  may  be  prudent  to  involve 
someone  from  this  community  early  in  the  process  of  developing 
an  evaluation  plan  to  encourage  a  team  effort.  Consideration 
should  also  be  given  during  construction  of  the  plan  to 
allowing  flexibility  in  the  event  both  parametric-based  and 
traditional  cost  proposals  are  submitted.  As  discussed  in 
question  6g,  fairness  is  essential  in  any  source  selection 
plan. 
E .    NEGOTIATIONS 

Question  ten  is  a  four  part  series  of  statements  dealing 
with  the  effects  that  parametric  cost  estimating  may  have  on 
the  negotiation  process.  This  process  can  be  a  factor  in  both 
pre-award  as  well  as  post-award  circumstances  as  noted  in 
question  four.  Issues  in  the  fourth  subsidiary  research 
question  are  addressed  in  this  series  of  statements. 

1.    Cost  or  Price  Negotiation  Criticality 

The  first  area  of  interest  in  statement  10a,  shown  below, 

is  the  effect  estimate  accuracy  might  play  in  negotiations. 

Using  parametric  cost  estimating  methods  should  make 
negotiating  costs  less  critical  as  cost  estimate  accuracy 
improves . 
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As  estimates  become  more  precise,  will  the  issue  of  cost 

become  less  of  an  issue?   The  question  assumes  that  cost  will 

always  remain  an  evaluation  factor  in  any  proposal  evaluation. 

There  was  a  lack  of  agreement  and  consensus  among  the 

participants  on  this  issue.    An  average  response  value 

indicates  disagreement  but  the  median  value  of  2  indicates 

agreement  with  the  statement.   The  program  staff  and  industry 

cost   analyst  data  displayed  in  Table   69  are  the  only 

demographic  categories  that  exhibit  any  consensus  on  this 

issue.    The  program  staff  group  is  unanimous  in  their 

disagreement  that  costs  will  become  less  important  as  estimate 

accuracy  improves.   The  industry  cost  analysts  see  a  shift  in 

emphasis  from  costs  to  other  factors  as  estimates  become  more 

precise. 

Table  69 
Question  10a  Group  Data 


Min 

Max 

M 

Median 

o 

Ql 

Q3 

Program 
Staff 

3.00 

3.00 

3.00 

3.00 

.00 

3.00 

3.00 

Industry 

Cost 

Analysts 

2.00 

3.00 

2.14 

2.00 

.38 

2.00 

2.00 

Source:  Developed  by  researcher. 

FAR  15.406-4  notes  that  cost  will  be  included  as  a  factor 
in  source  selection.  This  statement  raises  the  question  of 
how  much  will  cost  be  a  factor,  if  because  of  increasing 
accuracy,   the   acceptable   range   of   both   parties   in   a 
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negotiation  is  very  narrow  and  overlapping.    Parametric 

estimates  derived  from  the  same  models  and  estimating  systems 

for  both  offeror  and  buyer  have  the  potential  for  this 

situation  if  the  offeror's  facilities  are  used  as  discussed  in 

the  statement  8d  analysis.   In  the  event  that  this  occurs,  the 

negotiations  may  shift  from  cost  concerns  to  ones  of  a 

technical  ability  or  schedule  nature.   Another  possibility 

that  is  more  likely  is  a  shift  in  negotiations  to  inputs  to 

the  model  and  the  assumptions  that  go  with  them  if  the  model 

and  estimating  system  are  accepted  as  validated  by  both 

parties  in  a  negotiation  situation. 

2.    Cost  as  an  Independent  Variable  Impacts 

Statement  10b  attempts  to  relate  Cost  As  An  Independent 

Variable  (CAIV)  with  parametric  cost  estimating  during  the 

negotiation  process.   The  second  statement  of  question  ten  is 

shown  as: 

Using  parametric  cost  estimating  methods  should  have 
little  effect  on  negotiations  as  the  concept  of  cost  as 
an  independent  variable  shifts  emphasis  to  technical  and 
schedule  concerns  for  a  given  value. 

The  concept  of  CAIV  shifts  emphasis  to  technical  or  other 
issues  by  making  costs  more  of  a  constraint  than  an  outcome  of 
a  series  of  events  in  a  program.  Prior  to  CAIV,  there  was  an 
emphasis  on  cost  control  but  it  was  still  managed  as  a 
dependent  variable  in  the  program  life  cycle.   CAIV  introduced 
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the  idea  that  costs  would  actually  be  controlled  to  some 
degree  by  working  within  some  trade  space  with  technical 
capability,  logistics  supportability,  schedule,  and  other 
concerns  to  produce  and  field  affordable  weapons  systems  in  a 
timely  manner. 

There  was  a  large  degree  of  disagreement  with  this 
statement.  The  majority  of  respondents  to  this  question  (104) 
either  disagreed  or  strongly  disagreed  with  this  statement. 
The  disagreement  was  somewhat  of  a  consensus  with  an  average 
value  of  2.83,  a  standard  deviation  of  .70  and  a  median  value 
of  3.  Disagreement  with  this  statement  was  widespread 
throughout  the  DoD  acquisition  community  as  shown  by  the  group 
data  in  Table  70.  Four  of  the  largest  groups  of  participants, 
contracting  officers,  DCAA,  DCMC,  and  the  U.S.  Army  had  a 
consensus  in  their  disagreement  with  this  issue. 


Table  70 
Question  10b  Group  Data 


Min 

Max 

V 

Median 

a 

Ql 

Q3 

Contracting 
Officers 

2.00 

4.00 

2.91 

3.00 

.61 

3.00 

3.00 

U.S.  Army 

2.00 

4.00 

2.93 

3.00 

.66 

2.25 

3.00 

DCAA 

2.00 

4.00 

2.92 

3.00 

.66 

2.00 

3.00 

DCMC 

2.00 

4.00 

2.82 

3.00 

.66 

2.00 

3.00 

Source:  Developed  by  researcher 
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The  depth  of  feeling  over  this  indicates  that  CAIV  will  have 
little  effect  during  negotiations  in  shifting  emphasis  to 
other  issues. 

3.    Minor  Impacts  on  Source  Selection 

While  cost  realism  may  be  a  major  part  of  a  source 

selection  plan,  it  should  be  limited  in  its  impact  on  other 

areas  of  an  evaluation.   All  of  the  proposal  component  parts 

are  interrelated  to  some  degree  but  the  factors  should  have 

some  degree  of  independence.    This  statement  explores  the 

effects  of  parametric  cost  estimating  methods  and  cost  realism 

in  source  selection. 

Using  parametric  cost  estimating  methods'  should  have 
little  effect  on  source  selection,  they  only  affect  cost 
realism. 

The  aggregate  data  for  this  statement  shows  disagreement  with 

an  average  response  value  of  2.66,  a  standard  deviation  of 

.78,  and  a  median  value  of  3.   Table  71  contains  data  from  a 

major  buying  organization,  USAF,  and  two  vital  components  of 

the  DoD  acquisition  community,  the  auditor  and  cost  estimator 

groups,  that  disagree  with  this  statement. 

This  is  contrasted  with  the  regression  analysis  performed 

with  the  business  size  data.   The  equation  and  data 

Y  =  -.360 (X)  +7.38         R2  =  .18 
(.17)     (2.30) 
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display  a  negative  relationship  between  business  size  and  the 
response  value  for  this  statement.  As  the  business  size 
increases  so  does  the  level  of  agreement  with  this  statement. 
The  industry  and  corporate  policy  groups  do  not  show  any 
consensus  in  their  opinion  on  this  statement,  however. 


Table  71 
Question  10c  Group  Data 


Min 

Max 

M 

Median 

o 

Ql 

Q3 

Auditors 

1.00 

4.00 

2.56 

2.50 

.70 

2.00 

3.00 

Cost  Analysts 

2.00 

4.00 

2.87 

3.00 

.66 

2.00 

3.00 

USAF 

2.00 

4.00 

3.06 

3.00 

.68 

3.00 

3.75 

Source:  Developed  by  researcher. 

4 .    Significant  Impacts  on  Source  Selection 

The  interrelated  nature  of  the  parts  of  an  offeror's 

proposal  could  make  the  effects  of  one  element  have  a 

disproportionate  effect  in  the  source  selection  process. 

There  was  a  similar  lack  of  consensus  on  statement  lOd  (shown 

below)  as  there  was  in  statement  10c. 

Using  parametric  cost  estimating  methods  should  have  a 
major  effect  on  source  selection,  they  affect  past 
performance,  technical,  management,  and  cost  factors  in 
proposal  evaluation  schemes. 

An  average  response  value  of  2.48  and  a  median  value  of  2  was 
observed.   The  distribution  of  responses  was  strongly  agree, 
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20,   agree,   67,   disagree,   68,   and  strongly  disagree  16. 

Relatively  few  people  (39)  chose  no  opinion  or  did  not  respond 

to  this  statement.    The  DCAA  and  corporate  policy  group 

responses  in  Table  72  are  indicative  of  the  range  of  opinion 

to  this  statement.   They  are  not,  however,  indicative  of  the 

degree  of  consensus  on  this  issue.   These  two  groups  were  the 

only  ones  to  show  any  sort  of  consensus  of  the  demographic 

groups  analyzed.   The  contrasting  opinions  between  industry 

and  DC7AA  in  the  acquisition  community  is  clear.   The  industry 

business  size  regression  analysis  results 

Y  =  +.385(X)  -  2.77         R2  =  .22 
(.16)     (2.21) 

show  a  positive  relationship  between  the  responses  and  the 

size  of  the  business.   The  smaller  businesses  strongly  agree 

with  the  statement  while  the  larger  ones  merely  agree. 

Table  72 
Question  lOd  Group  Data 


Min 

Max 

M 

Median 

a 

Ql 

Q3 

Corporate 
Policy 

1.00 

2.00 

1.75 

2.00 

.50 

1.25 

2.00 

DC7AA 

2.00 

4.00 

2.81 

3.00 

.69 

2.00 

3.00 

Source:  Developed  by  researcher. 

The  industry  responses  to  this  statement  and  question  10c 
are  consistent.  The  regression  analysis  demonstrates  opposite 
trends  for  these  two  contrasting  statements,  and  from  this  one 
could  interpret   that   the   industry  portion  of   the   DoD 
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acquisition  community  views  a  proposal  as  a  highly  integrated 
whole  rather  than  separate  sections.  The  Government,  however, 
may  not  see  the  issue  in  the  same  way.  DCAA  responding  to 
this  statement  and  the  technical  support  group  in  responding 
to  question  nine  disagree  with  the  idea  that  using  parametric 
cost  estimating  methods  can  have  significant  effects  on  the 
source  selection  and  subsequent  negotiation  processes. 

Dennedy,  Deutsch,  and  Hertling  also  disagree  with  the 
premise  that  the  use  of  parametric  cost  estimating  methods  can 
have  an  effect  on  negotiations.  [Ref  8:p.3]  The  negotiation 
process  is  based  on  estimates  of  all  types  and  parametric 
estimates  are  only  one  of  many  sources  of  information.  In 
total,  the  results  of  statements  10a  through  lOd  indicate  that 
the  DoD  acquisition  community  does  not  believe  that  the  use  of 
parametric  cost  estimating  methods  will  change  the  fundamental 
nature  of  negotiations.  It  will  simply  redefine  some  of  the 
points  around  which  the  negotiations  may  flow. 
F.       SUMMARY 

This  chapter  and  the  preceding  two  chapters  described  the 
data  collection  efforts  and  analyses  conducted  for  this 
thesis.  Data  collection  was  chiefly  done  through  a  ten 
question  survey  distributed  to  members  of  the  DoD  acquisition 
community.  Personal  interview  information  was  also  collected 
and  presented  with  each  question's  survey  information  as  it 
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pertained  those  questions  or  statements.  The  combined  data 
were  examined  for  indications  regarding  the  overall  opinion  of 
the  entire  DoD  acquisition  community  about  each  issue.  The 
demographic  group  data  were  then  examined  and  contrasted  with 
the  entire  group  responses  and  between  group  responses.  Any 
groups  within  the  database  that  had  any  opinion  that  was 
markedly  different  from  that  of  the  entire  respondent  pool 
were  highlighted.  An  analysis  of  this  information  was 
presented  for  each  segment  of  each  question.  The  final 
chapter  in  this  thesis  will  conclude  the  study  with 
conclusions  and  recommendations  made  from  the  analysis. 
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IX.  PRINCIPAL  FINDINGS,  CONCLUSIONS,  AND  RECOMMENDATIONS 

A.  PURPOSE 

The  primary  purpose  of  this  study  was  to  examine  the 
effects  of  increasing  the  use  of  parametric  cost  estimation 
methods  in  individual  contract  actions.  This  chapter  will 
provide  conclusions  about  those  effects  and  recommendations 
for  expanding  the  part  that  parametric  cost  estimating  methods 
play  in  Federal  Government  acquisition  processes.  Principal 
findings  and  conclusions  will  be  addressed  and  then  they  will 
be  followed  by  the  recommendations.  Areas  of  further  research 
will  be  the  final  section  of  this  thesis. 

B.  CONCLUSIONS 

The  DoD  acquisition  community  views  the  use  of  parametric 
cost  estimating  as  having  limited  applicability.  Areas  that 
can  be  termed  traditional  uses  have  the  highest  degree  of 
utility  in  questions  one  through  four.  The  acquisition 
community  believes  that  the  methods  are  best  applied  to 
developmental  applications  and  serve  little  purpose  outside 
those  areas.  The  data  support  the  conclusion  that  the  DoD 
acquisition  community  is  not  familiar  with  the  techniques  and 
methods  used  in  parametric  cost  estimating.  They  do  not  have 
an   appreciation   of   the   benefits   that   parametric   cost 
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estimating  can  provide  across  the  entire  spectrum  of  activity 
in  the  DoD  acquisition  process. 

Practical  experience  with  using  the  methods  in  the 
acquisition  process  and  research  on  the  possible  applications 
for  the  methods  provide  a  different  view.  NASA,  industry,  and 
the  control  group  responses  clearly  showed  that  the  methods 
can  be  used  successfully  in  a  far  wider  range  of  activities 
than  they  are  normally  applied  by  DoD  agencies.  They  also 
demonstrated  areas  that  have  clearly  not  benefitted  by  the  use 
of  parametric  cost  estimating  methods,  such  as  contracting  for 
commodities,  T&M/Labor  hour  contract  situations,  and  in  most 
areas  of  the  PF/D&OS  phase.  Parametric  cost  estimating  also 
may  be  applied  in  those  areas,  however,  currently  they  are  not 
being  used  there.  The  methods  may  not  provide  enough 
improvement  over  existing  estimating  systems  to  make 
investments  in  them  cost  effective. 

The  potential  effects  of  using  parametric  cost  estimating 
methods  will  require  some  changes  to  be  made  to  the  way  data 
are  collected,  reviewed,  presented,  and  analyzed.  However, 
they  will  not  fundamentally  change  the  acquisition  process. 
The  DoD  acquisition  community  views  parametric  cost  estimating 
methods  as  an  incremental  step  in  increasing  the  efficiency  of 
the  acquisition  process.  The  community  acknowledges  the  fact 
that  the  methods  can  affect  almost  every  action  in  the  DoD 
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procurement  process.  They  also  note  that  if  the  methods  are 
to  be  used,  they  must  be  considered  from  the  initial  planning 
stages . 

The  members  of  the  acquisition  community  also  are  viewing 
the  use  of  parametric  cost  estimating  methods  in  terms  of 
acquisition  reform.  The  data  demonstrate  a  preference  for 
giving  the  contractor  community  latitude  to  use  the  methods  as 
they  fit  best  in  the  contractor' s  systems  rather  than  as 
requirements  to  be  met.  The  data  also  indicate  that  the 
acquisition  community  would  accept  the  idea  that  teaming 
concepts  have  potential  benefits  to  parametric  cost 
estimating.  Major  stakeholders  in  the  community  identified 
each  other  as  important  members  of  the  process  whereby  a 
parametric  cost  estimating  model  is  calibrated  and  validated 
for  use.  The  use  of  teaming  and  parametric  cost  estimating 
fit  in  well  with  the  concept  of  disengagement  by  DoD  in  its 
contractor  estimation  system  review  programs. 
C.    RECOMMENDATIONS 

The  primary  recommendation  of  this  thesis  is  to  maintain 
the  Parametric  Cost  Estimating  Initiative  and  the  Joint 
Industry/Government  Working  Group.  This  initiative,  through 
the  working  group,  has  been  instrumental  in  bringing  awareness 
to  the  benefits  that  parametric  cost  estimating  can  provide. 
The  workshops  which  the  working  group  sponsors  and  newsletters 
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that  it  publishes  are  outstanding  fora  for  publicizing  these 
benefits,  exploring  new  areas  in  which  the  methods  can  be 
applied,  and  producing  meaningful  discussion  about  actual 
application  of  the  methods  in  specific  situations.  The  large 
number  of  surveys  not  answered  indicates  that  the  majority  of 
the  DoD  acquisition  community  is  still  unfamiliar  with 
parametric  cost  estimating  methods. 

A  second  recommendation  is  to  evaluate  and  publish 
results  from  the  Joint  Government/ Industry  Parametric  Cost 
Estimating  Pilot  Program.  The  lessons  learned  and  successes 
will  help  to  promote  the  use  of  the  methods  and  also  to 
publicize  how  they  can  benefit  both  Government  and  Industry. 
Business  case  analyses  could  be  performed  with  these  results 
to  quantify  the  effects  of  using  parametric  cost  estimating 
methods  and  implementation  procedures. 

Training  the  acquisition  community  is  the  third 
recommendation  of  this  thesis.  One  of  the  major  efforts  of 
the  working  group  has  been  to  establish  a  training  program  in 
the  area  of  parametric  cost  estimating.  The  training  should 
be  made  readily  available  to  the  acquisition  community  as 
rapidly  as  possible  to  reap  the  maximum  benefits  from  the  use 
of  the  methods.  Training  should  also  include  a  review  of  the 
existing  curricula  for  acquisition  workforce  professionals. 
A  brief  discussion  regarding  parametric  cost  estimating 
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methods  to  introduce  them  to  the  workforce  could  provide  basic 
awareness  and  understanding  of  the  methods  if  it  were  added  to 
existing  cost  and  pricing  course  materials . 

Parametric  cost  estimating  methods  should  be  considered 
for  incorporation  in  DoD  modeling  and  simulation- programs  for 
acquisition  purposes.  The  U.S.  Navy  Acquisition  Center  for 
Excellence  (ACE)  could  potentially  benefit  from  the  use  of  the 
methods  as  well  as  provide  awareness  to  the  acquisition 
community  about  their  potential  applications.  The  U.S.  Navy 
Research  Development  and  Acquisition  Team  Strategic  Plan  for 
1996  and  1997  provides  several  other  areas  where  the  methods 
could  be  applied.  The  strategic  goals  for  total  ownership 
cost  and  innovation/technology  insertion  discuss  modeling  and 
simulation  areas  that  could  also  benefit  from  using  parametric 
cost  estimating  methods. 

Connectivity  between  databases  should  be  improved.  As 
more  of  the  DoD  systems  and  equipments  incorporate  COTS  and 
NDI,  integration  will  become  a  major  cost  driver.  Parametric 
estimating  could  be  helpful  in  modeling  the  integration 
schedule  and  costs  if  databases  for  the  individual  equipments 
could  be  integrated.  Parametric  estimation  of  total  life 
cycle  costs  could  also  benefit  from  greater  connectivity 
between  databases. 
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D.  FURTHER  RESEARCH  TOPICS 

The  following  is  a  list  of  several  areas  that  might  be  of 
interest  and  benefit  to  the  acguisition  community  regarding 
parametric  cost  estimating: 

1 .  Business  case  analyses  of  the  Government  agencies  and 
companies  involved  in  the  Joint  Government/ Industry 
Parametric  Cost  Estimating  Pilot  Program  could  provide 
greater  insights  to  the  application  of  parametric  cost 
estimating  methods  to  DoD  acquisition. 

2.  A  review  of  how  the  use  of  parametric  cost  estimating 
methods  affect  the  DoD  estimating  system  review  program 
would  be  useful  to  buying  organizations  as  well  as  DoD 
administrative  agencies. 

3.  Business  case  analyses  of  parametric  estimating 
applications  to  the  concept  of  total  life  cycle  costs 
would  provide  benefits  to  future  programs  and  DoD 
managers . 

4 .  A  review  of  the  regulatory  and  other  barriers  to 
effective  implementation  of  using  parametric  costing 
could  benefit  those  interested  in  promoting  the  wider  use 
of  the  methods. 

E.  FINAL  SUMMARY 

The  final  summary  of  the  issues  addressed  in  this  thesis 
will  be  a  restatement  of  the  research  questions  followed  by 
brief  answers. 
Primary  Research  Question: 

How  could  a  DoD  emphasis  on  using  parametric  methods  in 
cost  estimating  affect  the  procurement  process? 
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Answer:  A  DoD  emphasis  on  parametric  methods  in  cost 
estimating  could  influence  almost  every  step  in  the 
acquisition  process  in  some  way.   The  effects  that  they 
would  cause  would  be  minor  in  nature,  however. 
Subsidiary  Research  Questions: 

1.  What  changes  could  offerors  make  to  their  proposal 
preparation,  submission,  and  support  processes  to 
accommodate  using  parametric  methods,  techniques,  or 
software  in  estimating  potential  contract  costs? 
Answer:  Offerors  desiring  to  incorporate  parametric  cost 
estimating  methods  into  their  proposal  preparation 
processes  could  either  use  a  commercially  available  model 
or  develop  one  of  their  own.  In  either  case  a  database 
from  which  to  work  is  essential  and  must  be  constructed. 
The  decisions  made  here  will  influence  the  ability  of  the 
offeror  to  respond  successfully  to  Federal  Government 
solicitations . 

2 .  What  impacts  could  emphasis  on  using  cost  estimates 
based  on  parametric  estimating  techniques  have  on  source 
selection  criteria  and  on  the  evaluation  of  proposals  by 
DOD? 

Answer:  Cost  estimates  based  on  parametric  estimates 
could  impact  the  entire  source  selection  process.  They 
have  impacts  on  the  evaluation  plan,   the  proposals 
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submitted,  and  thus  the  material  available  for  evaluation 
and  source  selection  decisions. 

3.  How  could  source  selection  criteria  and  evaluation 
factors  be  constructed  to  fairly  evaluate  a  proposal  that 
uses  a  parametric  cost  estimate  with  one  that  does  not? 
Answer:  Source  selection  criteria  must  be  flexible  enough 
to  include  proposals  based  on  parametric  estimates  as 
well  as  ones  based  on  traditional  estimates.  Failing  to 
do  so  could  impact  the  level  of  competition  in  a  given 
procurement . 

4.  What  effects  could  using  parametric  cost  estimations 
have  on  negotiations? 

Answer:  Parametric  cost  estimating  methods  should  have 
little  effect  on  negotiations.  The  points  around  which 
one  negotiates  may  shift  to  inputs  to  the  model  rather 
than  specific  cost  elements  but  the  basics  of  negotiation 
will  remain  the  same.   Cost  will  always  be  a  factor. 
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APPENDIX  A.  DEFINITIONS 

Analogy  Model  -  a  model  that  estimates  costs  of  a  new  program 
or  system  from  data  on  past  costs  of  similar  programs  or 
systems.  [Ref  23:p.290] 

Analysis  of  Variance  -  an  expression  of  the  measure  of  total 
variation  in  a  data  set  as  a  sum  of  terms,  which  can  be 
attributed  to  specific  sources,  or  causes  of  variation.  [Ref 
37:p.207] 

Benchmarking  Process  Costs  -  generating  an  estimate  of  the 
costs  associated  with  a  process  for  later  comparison  with 
other  estimates  or  actual  costs. 

Bottoms  Up  Cost  Estimate  (Detailed  Cost  Estimate)  -  an 
estimate  of  project  component  costs  from  detailed 
understanding  of  the  cost  elements  of  each  component. 
Supplier  quotations,  labor  hour  estimates  from  time  and  motion 
studies  or  labor  hour  and  material  estimates  from  learning 
curve  analyses  form  the  basis  of  these  estimates.  [Ref 
2:p.294] 


201 


Calibration  -  the  process  of  indexing  a  parametric  model  or 
system  to  local  cost  and  product  history.  [Ref  5:p.4] 

Capacity  Cost  Estimates  -  estimates  based  on  logarithmic  plots 
of  similar  plants  which  show  a  relationship  between  capacity 
and  costs.  [Ref  14:p.l32] 

Coefficient  of  Determination  -  the  proportion  of  variation  in 
the  dependent  variable  that  has  been  explained  or  accounted 
for  by  the  regression  line.  [Ref  37:p.46] 

Commercial-Of f-The-Shelf  -  an  existing  system  or  equipment 
that  is  sold  to  the  Federal  Government  in  the  same 
configuration  that  it  is  sold  to  the  general  public  or  with 
minor  modifications.  [Ref  3: p. 42] 

Cost  Analysis  -  the  review  and  evaluation  of  a  contractor's 
costs  or  pricing  data,  and  of  the  judgmental  factors  applied 
in  projecting  from  the  data  to  the  estimated  costs,  for  the 
purpose  of  determining  the  degree  to  which  the  contractor's 
proposed  costs  represent  what  contract  performance  should 
cost,  assuming  reasonable  economy  and  efficiency.  [Ref  3:p  19] 
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Cost  or  Pricing  Data  -  all  facts  that  . . .  prudent  buyers  and 
sellers  would  reasonably  expect  to  affect  price  negotiations 
significantly.  [Ref  13:part  15.801] 

Cost  Realism  -  means  the  costs  in  an  offeror's  proposal  are 
realistic  for  the  work  to  be  performed,  reflect  a  clear 
understanding  of  the  requirements  and  are  consistent  with  the 
various  elements  of  the  offeror's  technical  proposed.  [Ref 
13:part  15.801] 

Dependent  Variable  -  the  variable  whose  value  is  to  be 
predicted.  [Ref  37:p.39] 

Equipment-Ratio  Estimates  -  estimates  derived  from  process 
flow  diagrams  and  equipment  lists  with  installation  and 
construction  costs  estimated  as  a  ratio  of  past  similar 
efforts.  [Ref  14:p.l32] 

Expert  Opinion  -  the  use  of  the  judgment  of  experts  when 
supporting  data  or  program  definitions  are  insufficient.  [Ref 
23:p.290] 

Independent  Variable  -  the  variable  about  which  knowledge  can 
be  obtained.  [Ref  37:p.39] 
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Layout  Estimates  -  estimates  from  plant  layouts  and  process 
flows  and  equipment  lists.  [Ref  14:p.l32] 

Learning  Curve  -  an  empirical  relationship  between  the  number 
of  units  of  an  item  produced  and  the  number  of  units  of  a 
resource  used  to  produce  them.  The  most  common  relationship 
used  is  units  of  production  and  number  of  labor  hours  required 
to  produce  the  units.  [Ref  l:p.267] 

Negotiation  -  a  process  between  buyers  and  sellers  seeking  to 
reach  mutual  agreement  on  a  matter  of  common  concern  through 
fact  finding,  bargaining,  and  persuasion.  [Ref  3:p.41] 

Non-developmental  Item  -  an  item  developed  exclusively  at 
private  expense  and  sold  in  substantial  quantities,  on  a 
competitive  basis,  to  multiple  state  and  local  governments. 
[Ref  13:part  2.101] 

Parametric  Cost   Estimate   -   an   estimate   derived  from 

statistical  correlation   of   historic   system   costs  with 

performance  and/or  physical  attributes  of  the  system.  [Ref 
37:p.l72] 
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Performance  Specification  -  a  description  of  a  deliverable  in 
terms  of  desired  operational  capabilities.  [Ref  3:p.44] 

Preliminary  Bill  Estimates  -  estimates  derived  from  bills  of 
material  produced  from  initial  drawings  and  labor-material 
ratios  from  past  work.  [Ref  14:p.l32] 

Price  Analysis  -  the  process  of  examining  and  evaluating  a 
proposed  price  without  evaluating  its  separate  cost  elements 
and  proposed  profit.  [Ref  13:part  15.8] 

Regression  Analysis  -  the  mathematical  nature  of  the 
association  between  two  variables.  [Ref  37:p.39] 

Risk  -  the  probability  of  not  attaining  the  goals  for  which 
the  party  entered  a  contract.  [Ref  3: p. 51] 

Source  Selection  -  the  process  wherein  the  requirements, 
facts,   recommendations  and  policies  relevant  to  an  award 
decision  in  a  competitive  procurement  of  a  system/project  are 
examined  and  the  decision  is  made.  [Ref  3:p.53] 
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Standard  Error  of  the  Estimate  -  the  standard  deviation  of  the 
population  of  y  values  predicted  from  a  single  value  of  x. 
[Ref  23:p.305] 

Top  Down  Cost  Estimate  -  an  overall  cost  estimate  for  a 
proposed  project  based  on  the  global  properties  of  the  project 
from  which  component  parts  are  partitioned  for  planning 
purposes.  [Ref  34:p.l76] 

Validation  -  the  process  or  act  of  demonstrating  the 
calibrated  model's  ability  to  function  as  a  credible  forward 
estimating  tool  or  system-  [Ref  5:p.4] 

Work  Breakdown  Structure  -  a  product-oriented  family  tree 
resulting  from  system  engineering  efforts  which  completely 
define  the  program.  [Ref  37:p.l76] 
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APPENDIX  B.  LIST  OF  ACRONYMS 

Ed2  -  Sum  of  the  Squares 

a2  -  Population  Variance 

ACE  -  Acquisition  Center  of  Excellence 

ACO  -  Administrative  Contracting  Officer 

ACT  -  Active  Coefficient 

CAIG  -  Cost  Analysis  Improvement  Group 

CAIV  -  Cost  As  An  Independent  Variable 

CAS  -  Cost  Accounting  Standards 

CE  -  Concept  Exploration 

CER  -  Cost  Estimating  Relationship 

CICA  -  Competition  in  Contracting  Act 

COCOMO  -  Constructive  Cost  Model 

COTS  -  Commercial  off  the  Shelf 

CPAF  -  Cost-Plus-Award-Fee  Contract 

CPFF  -  Cost-Plus-Fixed-Fee  Contract 

CPIF  -  Cost-Pius-Incentive-Fee  Contract 

CPRG  -  Contract  Pricing  Reference  Guides 

DCAA  -  Defense  Contract  Audit  Agency 

DCMC  -  Defense  Contract  Management  Command 

DEVC  -  Development  Cost  Variable 

DF  -  Degrees  of  Freedom 

DFARS  -  Defense  Federal  Acquisition  Regulation  Supplement 
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DLSIE  -  Defense  Logistics  Studies  Information  Exchange 

DoD  -  Department  of  Defense 

DoE  -  Department  of  Energy 

DoN  -  Department  of  Navy 

DTIC  -  Defense  Technical  Information  Center 

ECP  -  Engineering  Change  Proposal 

E&MD  -  Engineering  and  Manufacturing  Development 

F  -  F  Statistic 

FAR  -  Federal  Acquisition  Regulation 

FFP  -  Firm  Fixed-Price  Contract 

FPE  -  Fixed-Price  with  Economic  Price  Adjustment  Contract 

FPIF  -  Fixed-Price  Incentive  Firm  Contract 

FPIS  -  Fixed-Price  Incentive,  Successive  Targets  Contract 

FPRA  -  Forward  Pricing  Rate  Agreement 

FPR(P)  -  Fixed-Price  with  Prospective  Redetermination  Contract 

FPR(R)  -  Fixed-Ceiling-Price  with  Retroactive  Redetermination 

Contract 
ICE  -  Independent  Cost  Estimate 
GS  -  General  Service 

GSA  -  General  Services  Administration 
IPCE  -  Independent  Parametric  Cost  Estimate 
LG  -  Large  Ship  Factor 
LRIP  -  Low  Rate  Initial  Production 
LSBF  -  Least  Squares  Best  Fit 
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NASA  -  National  Aeronautics  and  Space  Administration 

NDI  -  Non-developmental  Item 

OSD  -  Office  of  the  Secretary  of  Defense 

P  -  P  value 

PCO  -  Procuring  Contracting  Officer 

PD&RR  -  Program  Definition  and  Risk  Reduction 

PF/DOS  -  Production,  Fielding/Deployment  and  Operational  Support 

PRICE  -  Programmed  Review  of  Information  for  Costing  and 

Evaluation 
PROC  -  Procurement  Cost  Variable 
R2  -  Coefficient  of  Determination 

RDT&E  -  Research,  Development,  Test,  and  Evaluation 
SES  -  Senior  Executive  Service 
SM  -  Small  Ship  Factor 
SOW  -  Statement  of  Work 
SPI  -  Single  Process  Initiative 
SSE  -  Standard  Sum  of  the  Errors 
T&M  -  Time  and  Materials  Contract 
TINA  -  Truth  in  Negotiations  Act 
USAF  -  United  States  Air  Force 
USA  -  United  States  Army 
USN  -  United  States  Navy 
WBS  -  Work  Breakdown  Structure 
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APPENDIX  C.  SURVEY  QUESTIONS 


Please  use  the  following  as  a  definition,  taken  from  the  DoD 
Parametric  Cost  Estimating  Handbook  published  in  the  fall  of 
1995,  for  Parametric  Cost  Estimating  (PCE) : 

"A  technique  employing  one  or  more  Cost  Estimating 
Relationships  (CERs)  and  associated  mathematical  relationships 
and  logic.  The  technique  is  used  to  measure  and/or  estimate 
the  cost  associated  with  the  development,  manufacture,  or 
modification  of  a  specified  end  item.  The  measurement  is 
based  on  the  technical,  physical,  or  other  end  item 
characteristics . " 

PARAMETRIC  COST  ESTIMATING  SURVEY 
Please  complete  the  following  demographic  information. 

1.  What  is  your  job  classification?  (Select  1  that  fits  best  or  write  in  a  specific  title  if  none  apply.) 
Cost  Estimator  Cost/Price  Analyst  Contract  Negotiator 

Contracting  Officer  Program  Staff  Program  Manager  Engineer 

Legal  Support  Technical  Support     Auditing     Accounting 

Other  

2.  If  employed  by  industry,  annually  does  your  company  do  business  with  the  Federal 
Government  totaling: 

<  $100  Million  $100  -  500  Million       $  500  -  $  1  Billion       >  $1  Billion 


3.  If  employed  by  the  Federal  Government: 

USA  DCMC  DCAA  NASA  USN  USAF  GSA 

Other: GS/SES  Level/Rank 

4.  How  long  have  you  been  working  in  the cost  estimating or  acquisition  fields? 

(Please  select  either  career  field  and  the  appropriate  length  of  time.) 

One  year  or  less    1  -5  years    5-10  years 1 0-20  years Over  20  years 

5.  What  functions  involving  cost  estimating  do  you  primarily  perform? 

Prepare  Estimates  Use  in  Negotiation  Preparation  or  Proposal  Analysis 

Review/ Audit  Estimates        Set Corporate  or Federal  Government  Policy 

6.  What  is  your  level  of  mathematics  knowledge? 

High  School  Algebra/Calculus  College  Algebra   College  Business  Calculus 

College  Engineering  Calculus  Series   College  Mathematics  Minor/Major 

Advanced  Degree  Involving  Training  in  Mathematics     Other 
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Please  record  your  opinions  regarding  the  following  questions  relating  to  PCE. 

For  questions  1  through  4,  use  the  following  scale  to  represent  the  utility  of  PCE  in  the  listed 

areas. 
(1)  Vital     (2)  Very  Useful     (3)  Somewhat  Useful      (4)  No  Use      (5)  Hinders  the  Process 
1 .  How  useful  is  PCE  in  these  areas? 

Bench  Marking  Process  Costs    Validating  Bottom's  Up  Estimates 

Program  Estimates  Budgeting  ICE  Generation   Software  Contracts 

Repair  Parts  Contracts  R&D  Contracts    Service  Contracts 

Hardware  Contracts  Construction  Contracts    Commodity  Contracts 


PCE  is  also  useful  for:  (Please  fill  in  other  areas.) 

2.  How  useful  is  PCE  in  each  of  the  stages  of  a  system's  life  cycle? 
Concept     Program  Definition   Engineering  &  Manufacturing    Production,  Fielding/ 
Exploration       &  Risk  Reduction  Development  Deployment  &  Operational 

Support 


3.  How  useful  is  PCE  in  situations  where  you  would  also  use  the  following  contract  type: 
FTP     FPE     FPIS    FPR(P)    FPR(R)     FPIF  CPIF    CPAF    CPFF    T&M     Labor  Hour 


PCE  is  also  useful  for  this  contract  type  situation  .  (Please  fill  in  others.) 

4.  How  useful  is  PCE  in  estimating  costs  for  activities  in  these  two  areas  of  contract 
management? 

Pre- Award  Actions  Contract  Post- Award  Actions 

This  part  is  a  series  of  statements  about  the  use  of  PCE  methods.  Using  the  following  scale, 
please  indicate  to  the  left  of  each  statement,  the  degree  to  which  you  agree  or  disagree  with 
them. 
(1)  Strongly  Agree      (2)  Agree      (3)  Disagree      (4)  Strongly  Disagree      (5)  No  Opinion 

5.  PCE  methods,  models  and  software: 

a. Could  eventually  replace  traditional  "bottoms  up"  cost  estimating  methods  in  many 

acquisition  estimating  applications  as  a  result  of  acquisition  streamlining, 
b. Could  replace  traditional  cost  estimating  methods  as  larger,  more  compatible  data  bases, 

greater  computing  power  and  improved  software  become  available, 
c. Can  save  contractors  and  Federal  Government  oversight  officials  significant  time  and 

expense  in  developing  and  then  evaluating  proposals, 
d. Could  significantly  change  the  way  potential  contractors  collect  and  summarize  cost 

information. 

e. Should  decrease  the  probability  of  cost  overruns  in  programs  where  used. 

f. Should  change  the  way  Cost  and  Pricing  Data  submitted  to  support  a  proposal  is  viewed. 

6.  In  solicitations,  buying  activities  should: 

a. Encourage  that  PCE  be  used  for  specified  procurement  actions. 

b. Specify  PCE  methods,  models,  or  software  for  cost  proposal  construction. 

c. Leave  PCE  methods,  models,  or  software  selection  to  contractor  discretion. 

d. Require  a  specific  cost  breakdown  format  for  PCE  based  cost  proposals. 
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e. Require  a  specific  format  to  extract  labor  hour,  subcontractor,  or  material  quantity 

estimates  for  technical  proposal  evaluation, 
f. State  criteria  to  be  used  to  evaluate  the  statistical  significance  of  CERs  used  to  support 

a  PCE  model, 
g. Require  cost  proposals  based  on  either  PCE  methods  or  traditional  costing  methods. 

Evaluation  schemes  cannot  fairly  distinguish  between  proposals  using  different  costing 

methods  for  the  same  work. 

7.  A  potential  contractor  should  provide  the  following  to  support  a  cost  proposal: 
a. PCE  model  software  and  input  data  used. 

b. Justification  to  support  the  use  of  PCE  models  and  input  data. 

c. Data  and  rationale  used  to  validate  a  PCE  model  for  a  specific  set  of  circumstances. 

8.  DCAA  should  perform  the  following  activities:  (If  not  DCAA,  who  should  do  them?) 

a. Establish  metrics  to  determine  a  PCE  model's  validity  in  a  given  situation. 

b. Establish  criteria  to  determine  when  a  PCE  model  should  be  updated. 


c. Verify  that  a  PCE  model  accurately  predicts  costs  for  the  expected  contract  work. 

d. "Run"  a  contractor's  PCE  model  for  purposes  of  analyzing  the  cost  proposal. 

9. Construction  of  CERs  used  in  PCE  based  proposals  should  be  used  as  a  factor  in 


evaluating  a  potential  contractor's  level  of  understanding  of  the  technical  requirements  in  a 
solicitation. 

10.  Using  PCE  methods: 

a. Should  make  negotiating  costs  less  critical  as  cost  estimate  accuracy  improves. 

b. Should  have  little  effect  on  negotiations  as  the  concept  of  Cost  as  an  Independent 

Variable  shifts  emphasis  to  technical  and  schedule  concerns  for  a  given  value. 

c. Should  have  little  affect  on  source  selection,  they  only  affect  a  proposal's  cost  realism. 

d. Should  have  a  major  affect  on  source  selection,  they  affect  past  performance,  technical, 

management,  and  cost  factors  in  proposal  evaluation  schemes. 
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APPENDIX  D.  INTERVIEW  QUESTIONNAIRE 

l.How  should  an  emphasis  on  using  parametric  methods  in  cost 
estimating  affect  the  procurement  process? 

2. How  could  the  use  of  parametric  cost  estimates  be  incorporated 
into  program  management  beyond  their  traditional  uses? 

3.  How  far  should  DoD/DoN  go  in  emphasizing  parametric  cost 
estimating  methods? 

4. Should  the  use  of  parametric  cost  estimates  be  required  for 
certain  procurement  actions? 

5. What  information  should  DoD/DoN  agencies  expect  potential 
contractors  to  provide  in  support  of  their  proposals  that  use 
parametric  cost  estimating  techniques  for  proposal 
preparation? 

6. Do  you  see  any  major  changes  to  the  way  solicitations  are 
structured,  proposals  are  generated,  proposals  evaluated,  and 
sources  selected  because  of  an  emphasis  on  using  parametric 
cost  estimating  techniques? 

7. What  support  should  DoD/DoN  organizations  expect  of  DCAA  and 
DCMC  for  analyzing  parametric  cost  estimate  based  proposals? 

8 . Should  DoD/DoN  provide  guidance  on  how  the  variance  between  a 
parametric  cost  estimating  model  and  the  work  described  on  a 
solicitation  be  measured? 
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APPENDIX  E.  COLLECTED  DATA 


DEMOGRAPHIC  GROUP  INFORMATION 


Group  Name 

Sort 
Code 

#of 

Responses/ 
%of 
Responses 

Math 

Sort 

Code 

Math 
Sort 
Code 
a 

Experience 
Sort  Code 

Experience 
Sort  Code 
a 

Control  Group 

11        5.3% 

Job  Classification 

Cost  Estimator 

1 

14       6.7% 

38.5 

1.3 

28.1 

.89 

Cost/Price  Analyst 

2 

55     26.2% 

37.6 

1.43 

28.3 

.68 

Contract  Negotiator 

3 

8        3.8% 

36.9 

1.46 

27.5 

1.00 

Contracting  Officer 

4 

49     23.3% 

37.1 

1.47 

28.5 

.56 

Program  Staff 

5 

5        2.4% 

37.8 

1.47 

28.6 

.49 

Program  Manager 

6 

7        3.3% 

38.4 

1.18 

21.9 

.99 

Engineer 

7 

12       5.7% 

38.8 

1.09 

28.1 

.90 

Legal  Support 

8 

0 

Technical  Support 

9 

5         2.4% 

38.6 

1.20 

28.2 

.75 

Auditor 

10 

46      21.2% 

36.5 

.95 

28.6 

.53 

Accounting 

11 

2             1% 

38.0 

2.00 

27.5 

.5 

Other 

7          3.3% 

Industry  Size 

<$100  Million 

12 

2             1% 

38.0 

1.00 

28.0 

$100 -$500  Million 

13 

8          3.8% 

38.6 

1.26 

28.2 

.79 

$500  Million  -  $1  Billion 

14 

1            .5% 

39.0 

>  $1  Billion 

15 

9          4.3% 

39.0 

1.00 

28.3 

.66 

Total  Industry 

20         10% 
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Group  Name 

Sort 
Code 

#of 
Responses 

Math 

Sort 

Code 

Math 
Sort 
Code 
a 

Experience 
Sort  Code 

Experience 
Sort  Code 
a 

Government 

U.S.  Army 

16 

52      24.9% 

37.7 

1.59 

28.4 

.72 

DCMC 

17 

35      16.8% 

37.1 

1.36 

26.1 

.73 

DCAA 

18 

49      22.9% 

36.6 

1.06 

28.6 

.58 

NASA 

19 

9          4.4% 

38.0 

1.41 

28.0 

1.00 

U.S.  Navy 

20 

22      10.6% 

37.3 

1.33 

28.2 

.89 

USAF 

21 

17        8.2% 

37.9 

1.49 

28.2 

.76 

GSA 

22 

1            .6% 

37.0 

28.0 

OSD 

2             1% 

DoE 

2             1% 

Total  Government 

189       90% 

Level/Rank 

GS  12/03 

7          3.7% 

GS  13/04 

19        9.1% 

GS  14/05 

24      11.4% 

GS  15/06 

25      11.5% 

Experience 

Cost  Estimating 
Community 

23 

40      19.1% 

37.2 

1.43 

28.0 

.81 

Acquisition  Community 

24 

102    48.6% 

38.0 

1.54 

28.5 

.50 

One  Year  or  Less 

25 

0 

One  to  Five  Years 

26 

5          2.3% 

37.4 

.80 

Five  to  Ten  Years 

27 

19        9.1% 

37.8 

1.54 

Ten  to  Twenty  Years 

28 

85      40.5% 

37.6 

1.57 
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Group  Name 

Sort 
Code 

#of 
Responses 

Math 

Sort 

Code 

Math 
Sort 
Code 
o 

Experience 
Sort  Code 

Experience 
Sort  Code 
a 

Over  Twenty  Years 

29 

97      46.2% 

37.2 

1.41 

Work  Description 

Prepare  Estimates 

30 

37      17.6% 

38.1 

.70 

28.1 

.91 

Negotiation  Preparation 

31 

101    48.1% 

37.2 

1.46 

28.3 

.73 

Review/ Audit  Estimates 

32 

103    49.1% 

37.4 

1.52 

28.4 

.65 

Set  Corporate  Policy 

33 

2.4% 

39.0 

1.26 

28.0 

.63 

Set  Government  Policy 

34 

28      13.3% 

38.2 

1.54 

28.3 

.76 

Mathematics  Knowledge 

High  School 
Algebra/Calculus 

35 

24      11.4% 

28.3 

.55 

College  Algebra 

36 

64      30.5% 

28.5 

.58 

College  Business  Calculus 

37 

69      32.9% 

28.2 

.80 

College  Engineering 
Calculus 

38 

34      16.2% 

28.2 

.82 

College  Mathematics 
Major  or  Minor 

39 

31      14.8% 

28.6 

.67 

Advanced  Degree  With 
Mathematics 

40 

28      13.3% 

28.1 

.65 

SURVEY  DISTRIBUTION  AND  RETURN  DATA 


Group 
Name 

#  Surveys 
Sent 

#  Incorrect 
Addresses 

#  Surveys 
Returned 

%of 

Surveys 

Sent 

Group 

Response 

Rate 

%  of  Surveys 
Returned 

U.S.  Army 

332 

4 

52 

26.1 

15.9 

24.9 

DCMC 

203 

0 

35 

16.1 

17.2 

16.8 
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Group 
Name 

#  Surveys 
Sent 

#  Incorrect 
Addresses 

#  Surveys 
Returned 

%of 

Surveys 

Sent 

Group 

Response 

Rate 

%  of  Surveys 
Returned 

DCAA 

525 

131 

49 

31.2 

12.2 

22.9 

NASA 

39 

0 

9 

3.0 

23.1 

4.4 

U.S.  Navy 

120 

0 

22 

9.5 

18.3 

10.6 

USAF 

80 

10 

17 

5.6 

24.3 

8.2 

GSA 

9 

0 

1 

.7 

11.1 

.6 

DoE 

55 

0 

2 

4.4 

3.6 

1 

OSD 

11 

0 

2 

.9 

18.2 

1 

Industry 

33 

0 

20 

2.5 

64.5 

9.6 

Total 

1407 

145 

209 

100 

Avg  16.5 

100 
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TOTAL  GROUP  DATA 


Responses 

No 
Ans 

#1 

2 

3 

4 

5 

Avg 

Std 
Dev 

Var 

Med 

Skew 

QUESTION  1: 

Bench  Marking  Process  Costs 

76 

14 

42 

56 

21 

1 

2.66 

89 

.75 

3.00 

-0.14 

Validating  Bottom's  Up  Estimates 

50 

17 

52 

71 

17 

3 

2.61 

88 

78 

3.00 

-0.22 

Program  Estimates 

54 

29 

64 

56 

7 

0 

2.25 

.82 

.81 

2.00 

-0.21 

Budgeting 

48 

37 

68 

48 

9 

0 

2.20 

.85 

60 

2.00 

-0.35 

ICE  Generation 

90 

25 

41 

39 

15 

0 

234 

.95 

48 

2.00 

-0.37 

Software  Contracts 

71 

27 

29 

54 

27 

2 

2.54 

1.06 

.84 

3.00 

-0.59 

Repair  Parts 

66 

8 

34 

72 

30 

0 

2.85 

.80 

1.27 

3.00 

0.00 

R&D 

57 

18 

41 

49 

40 

5 

2.83 

1.06 

.76 

3.00 

-0.56 

Service 

58 

8 

29 

57 

57 

1 

3.09 

89 

77 

300 

-0.79 

Hardware 

47 

29 

56 

59 

19 

0 

2.55 

.85 

.85 

3.00 

-0.64 

Construction 

79 

11 

41 

54 

25 

0 

2.71 

.88 

48 

3.00 

-0.65 

Commodity 

101 

3 

13 

51 

41 

1 

3.23 

78 

.64     ■ 

300 

065 

QUESTION  2: 

CE 

45 

39 

48 

43 

33 

2 

2.46 

1.09 

1.27 

2.00 

0.89 

PDR&R 

40 

30 

60 

58 

21 

1 

2.43 

.95 

.80 

2.00 

0.43 

E&MD 

40 

22 

62 

76 

10 

0 

2.45 

.82 

.41 

300 

-0.59 

PFD&OS 

42 

19 

52 

70 

27 

0 

2.65 

.89 

.48 

300 

-0.05 

QUESTION  3: 

FFP 

52 

19 

55 

60 

23 

1 

2.56 

.90 

1.22 

300 

0.60 

FPE 

81 

12 

38 

53 

25 

1 

2.71 

90 

1.22 

3.00 

0.60 

FPIS 

83 

11 

42 

59 

15 

0 

2.62 

.81 

1.04 

3.00 

1.02 

FPR(P) 

77 

12 

46 

60 

15 

0 

2.60 

.81 

1.04 

3.00 

1.02 

FPR(R) 

79 

13 

42 

57 

19 

0 

2.64 

.85 

1.04 

3.00 

1.02 

FPIF 

69 

14 

52 

57 

18 

0 

2.56 

83 

1.22 

3.00 

0.60 

CPIF 

65 

19 

43 

59 

24 

0 

2.62 

91 

.92 

3.00 

0.54 

CPAF 

68 

20 

43 

55 

23 

1 

2.59 

95 

.92 

3.00 

0.54 

CPFF 

65 

22 

41 

59 

22 

1 

2.58 

95 

.92 

3.00 

0.54 

T&M 

69 

7 

26 

53 

49 

6 

3.13 

97 

50 

3.00 

0.94 

LABOR  HOUR 

73 

5 

30 

47 

49 

6 

3  16 

93 

56 

300 

0.62 
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No 
Ans 

#1 

2 

3 

4 

5 

Avg 

Std 
Dev 

Var 

Med 

Skew 

QUESTION  4: 

PRE-AWARD 

32 

32 

64 

69 

13 

0 

2.63 

.86 

.60 

2.00 

-0.35 

POST-AWARD 

41 

17 

32 

59 

53 

8 

3.04 

1.05 

1.26 

3.00 

-0.50 

QUESTION  5: 

A 

0 

28 

62 

67 

26 

27 

2.49 

.92 

.61 

3.00 

-0.18 

B 

0 

20 

82 

56 

15 

37 

2.37 

.81 

.43 

2.00 

-1.32 

C 

0 

45 

97 

37 

8 

23 

2.04 

.79 

.55 

2.00 

0.00 

D 

0 

34 

103 

35 

11 

27 

2.15 

.83 

1.17 

2.00 

0.79 

E 

0 

7 

43 

71 

39 

50 

2.89 

.82 

.54 

3.00 

-0.81 

F 

0 

18 

99 

41 

13 

39 

2.29 

.76 

.44 

2.00 

-0.43 

QUESTION  6: 

A 

0 

27 

95 

40 

12 

36 

2.21 

.78 

.69 

2.00 

0.69 

B 

0 

14 

49 

84 

29 

34 

2.73 

.83 

.74 

3.00 

-0.34 

C 

0 

22 

89 

40 

18 

41 

2.33 

.84 

.56 

2.00 

0.00 

D 

0 

24 

84 

55 

13 

34 

2.35 

.80 

1.29 

2.00 

0.23 

E 

0 

26 

77 

56 

15 

36 

2.37 

.83 

1.24 

2.00 

0  50 

F 

0 

36 

101 

28 

9 

36 

2.06 

.76 

.32 

2.00 

-0.77 

G 

0 

13 

66 

60 

20 

51 

2.55 

.80 

89 

3.00 

-0.89 

QUESTION  7: 

A 

0 

66 

95 

17 

7 

25 

1.81 

.76 

.51 

2.00 

0.33 

B 

0 

64 

107 

13 

3 

23 

1.75 

.65 

.23 

2.00 

-0.66 

C 

0 

71 

106 

6 

3 

24 

1.67 

.61 

.23 

2.00 

-0.66 

QUESTION  8: 

A 

0 

31 

90 

35 

19 

35 

2.25 

.87 

.69 

2.00 

-0.36 

B 

0 

32 

93 

33 

16 

36 

2.20 

.84 

.94 

2.00 

-0.20 

C 

0 

50 

102 

19 

8 

31 

1.93 

.75 

1.23 

2.00 

0.35 

D 

0 

30 

99 

24 

9 

48 

2.08 

.74 

.94 

2.00 

0.23 

QUESTION  9: 

0 

15 

82 

37 

13 

63 

2.37 

.83 

.03 

2.00 

-0.58 

QUESTION  10: 

A 

0 

10 

80 

66 

22 

32 

2.57 

.80 

.69 

2.00 

-0.41 

B 

0 

1 

48 

84 

20 

57 

2.83 

.70 

.98 

3.00 

0.51 

C 

0 

9 

66 

78 

21 

36 

2.66 

.78 

.56 

3.00 

0.57 

D 

0 

20 

67 

68 

16 

39 

2.48 

.85 

.65 

2.00 

-0.65 
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Control  Group  Data 


No  Response 

#1 

#2 

#3 

#4 

#5 

Std  Dev 

Variation 

Median 

Skew 

Question  1 

Program  Est 

0 

8 

1 

2 

0 

0 

1.45 

0.78 

0.61 

1 

1.50 

Budgeting 

0 

5 

4 

1 

1 

0 

1.82 

0.94 

0.88 

2 

1.20 

R&D 

1 

6 

2 

2 

0 

0 

1.60 

0.80 

0.64 

1 

0.61 

Software 

2 

0 

5 

2 

1 

1 

2.78 

1.03 

1.06 

2 

0.10 

Repair  Parts 

1 

4 

2 

4 

0 

0 

2.00 

0.89 

0.80 

2 

(0.15) 

Service 

2 

1 

2 

1 

3 

2 

3.33 

1.33 

1.78 

4 

(0.33) 

Hardware 

2 

2 

5 

2 

0 

0 

2.00 

0.67 

0.44 

2 

(0.45) 

Construction 

3 

2 

2 

3 

1 

0 

2.38 

0.99 

0.98 

3 

0.07 

Commodity 

5 

0 

0 

2 

1 

3 

4.17 

0.90 

0.81 

5 

0.08 

Question  2 

CE 

0 

7 

3 

1 

0 

0 

1.45 

0.66 

0.43 

1 

1.32 

PD&RR 

0 

5 

6 

0 

0 

0 

1.55 

0.50 

0.25 

2 

(0.21) 

E&MD 

0 

4 

4 

3 

0 

0 

1.91 

0.79 

0.63 

2 

0.19 

PF/D&OS 

0 

4 

1 

2 

4 

0 

2.55 

1.30 

1.70 

3 

(0.12) 

Question  3 

FFP 

2 

4 

3 

1 

0 

1 

2.00 

1.25 

1.56 

2 

1.28 

FPE 

4 

3 

2 

1 

0 

1 

2.14 

1.36 

1.84 

2 

1.36 

FPIS 

5 

3 

1 

2 

0 

0 

1.83 

0.90 

0.81 

2 

0.89 

FPR(P) 

5 

3 

1 

2 

0 

0 

1.83 

0.90 

0.81 

2 

0.89 

FPR(R) 

5 

3 

2 

1 

0 

0 

1.67 

0.75 

0.56 

2 

0.86 

FPIF 

3 

4 

3 

1 

0 

0 

1.63 

0.70 

0.48 

2 

0.35 

CPIF 

3 

4 

4 

0 

0 

0 

1.50 

0.50 

0.25 

2 

(0.19) 

CPAF 

3 

4 

4 

0 

0 

0 

1.50 

0.50 

0.25 

2 

(0.19) 

CPFF 

3 

5 

3 

0 

0 

0 

1.38 

0.48 

0.23 

1 

0.00 

T&M 

4 

1 

1 

2 

1 

2 

3.29 

1.39 

1.92 

3 

0.29 

Labor  Hour 

5 

1 

1 

1 

0 

3 

3.50 

1.61 

2.58 

4 

0.61 

Question  4 

Pre-Award 

2 

9 

0 

0 

0 

0 

1.00 

0.00 

0.00 

1 

(1.92) 

Post-Award 

3 

7 

1 

0 

0 

0 

1.13 

0.33 

0.11 

1 

0.03 
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No  Response 

#1 

#2 

#3 

#4 

#5 

Std  Dev 

Variation 

Median 

Skew 

Question  5 

5a 

0 

7 

2 

2 

0 

0 

1.55 

0.78 

0.61 

1 

1.15 

5b 

11 

0 

0 

0 

0 

0 

0.00 

0.00 

0.00 

0 

0.00 

5c 

0 

8 

3 

0 

0 

0 

1.27 

0.45 

0.20 

1 

1.19 

5d 

0 

3 

5 

1 

0 

2 

2.36 

1.37 

1.87 

2 

1.21 

5e 

11 

0 

0 

0 

0 

0 

0.00 

0.00 

0.00 

0 

0.00 

5f 

11 

0 

0 

0 

0 

0 

0.00 

0.00 

0.00 

0 

0.00 

Question  6 

6a 

0 

0 

4 

4 

3 

0 

2.91 

0.79 

0.63 

3 

0.19 

6b 

0 

0 

1 

7 

3 

0 

3.18 

0.57 

0.33 

3 

(0.03) 

6c 

11 

0 

0 

0 

0 

0 

0.00 

0.00 

0.00 

0 

0.00 

Sd 

1 

1 

1 

6 

2 

0 

2.90 

0.83 

0.69 

3 

(1.23) 

5e 

1 

1 

3 

4 

2 

0 

2.70 

0.90 

0.81 

3 

(0.69) 

6f 

0 

1 

6 

3 

1 

0 

2.36 

0.77 

0.60 

2 

0.54 

6g 

11 

0 

0 

0 

0 

0 

0.00 

0.00 

0.00 

0 

0.00 

Question  7 

7a 

0 

3 

5 

1 

2 

0 

2.18 

1.03 

1.06 

2 

0.74 

7b 

0 

4 

6 

1 

0 

0 

1.73 

0.62 

0.38 

2 

0.29 

7c 

0 

4 

6 

1 

0 

0 

1.73 

0.62 

0.38 

2 

0.29 

Question  8 

8a 

0 

0 

5 

4 

2 

0 

2.73 

0.75 

0.56 

3 

0.57 

8b 

0 

2 

7 

2 

0 

0 

2.00 

0.60 

0.36 

2 

0.00 

8c 

0 

4 

2 

5 

0 

0 

2.09 

0.90 

0.81 

2 

(0.21) 

8d 

11 

0 

0 

0 

0 

0 

0.00 

0.00 

0.00 

0 

0.00 

Question  9 

9 

0 

1 

4 

3 

2 

1 

2.82 

1.11 

1.24 

3 

0.42 

Question  10 

10a 

0 

1 

5 

2 

1 

2 

2.82 

1.27 

1.60 

2 

0.71 

10b 

11 

0 

0 

0 

0 

0 

0.00 

0.00 

0.00 

0 

0.00 

10c 

1 

2 

6 

2 

0 

0 

2.00 

0.63 

0.40 

2 

(0.69) 

10d 

1 

0 

2 

5 

3 

0 

3.10 

0.70 

0.49 

3 

(1.42) 
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Cost  Estimator  Group  Data 


No  Response 

#1 

#2 

#3 

#4 

#5 

Mean 

StdDev 

Variation 

Median 

Skew 

Question  1 

Benchmarking 

4 

2 

3 

3 

2 

0 

2.50 

1.02 

1.05 

3 

0.09 

Validating 

0 

1 

6 

5 

1 

1 

2.64 

0.97 

0.94 

3 

0.86 

Program  Est 

2 

3 

4 

4 

1 

0 

2.25 

0.92 

0.85 

2 

(0.15) 

Budgeting 

1 

4 

5 

4 

0 

0 

2.00 

0.78 

0.62 

2 

(0.31) 

ICE 

3 

1 

4 

5 

1 

0 

2.55 

0.78 

0.61 

3 

(0.49) 

Software 

3 

2 

2 

4 

2 

1 

2.82 

1.19 

1.42 

3 

(0.03) 

Repair  Parts 

5 

1 

1 

1 

6 

0 

3.33 

1.05 

1.11 

4 

(0.16) 

R&D 

2 

3 

2 

6 

1 

0 

2.42 

0.95 

0.91 

3 

(0.42) 

Service 

3 

1 

1 

5 

4 

0 

3.09 

0.90 

0.81 

3 

(070) 

Hardware 

2 

3 

5 

2 

0 

0 

1.90 

0.95 

0.91 

3 

(0.46) 

Construction 

3 

1 

1 

8 

1 

0 

2.82 

0.72 

0.51 

3 

(0-87) 

Commodity 

7 

1 

1 

4 

1 

0 

2.71 

0.88 

0.78 

3 

0.46 

Question  2 

CE 

0 

6 

5 

1 

2 

0 

1.93 

1.03 

1.07 

2 

1.04 

PD&RR 

1 

3 

6 

2 

2 

0 

2.23 

0.97 

0.95 

2 

0.20 

E&MD 

1 

1 

7 

5 

0 

0 

2.31 

0.61 

0.37 

2 

(1.14) 

PFD&OS 

1 

0 

3 

7 

3 

0 

3.00 

0.63 

0.40 

3 

(1.52) 

Question  3 

FFP 

7 

2 

3 

0 

2.86 

1.12 

1.27 

3 

0.64 

FPE 

7 

2 

3 

0 

2.86 

1.12 

1.27 

3 

0.64 

FPIS 

8 

2 

2 

0 

2.67 

1.11 

1.22 

3 

1.00 

FPR(P) 

8 

2 

2 

0 

2.67 

1.11 

1.22 

3 

1.00 

FPR(R) 

8 

2 

2 

0 

2.67 

1.11 

1.22 

3 

1.00 

FPIF 

7 

2 

3 

0 

2.86 

1.12 

1.27 

3 

0.64 

CPIF 

7 

2 

2 

2 

0 

2.71 

1.03 

1.06 

3 

0.63 

CPAF 

7 

2 

2 

2 

0 

2.71 

1.03 

1.06 

3 

0.63 

CPFF 

7 

2 

2 

2 

0 

2.71 

1.03 

1.06 

3 

0.63 

T&M 

8 

0 

1 

2 

3 

0 

3.33 

0.75 

0.56 

4 

0.56 

Labor  Hour 

7 

0 

2 

2 

3 

0 

3.14 

0.83 

0.69 

3 

0.37 

Question  4 

Pre-Award 

1 

4 

4 

5 

0 

0 

2.08 

0.83 

0.69 

2 

(0.38) 

Post-Award 

1 

2 

2 

3 

5 

1 

3.08 

1.18 

1.40 

3 

(0.56) 
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Question  5 

No  Response 

#1 

#2 

#3 

#4 

#5 

Mean 

Std  Dev 

Variation 

Median 

Skew 

5a 

0 

1 

5 

7 

1 

0 

2.57 

0.73 

0.53 

3 

(0.28) 

5b 

0 

1 

5 

8 

0 

0 

2.50 

0.63 

0.39 

3 

(0.98) 

5c 

0 

4 

5 

5 

0 

0 

2.07 

0.80 

0.64 

2 

(0.14) 

5d 

0 

4 

6 

2 

2 

0 

2.14 

0.99 

0.98 

2 

0.67 

5e 

2 

0 

3 

5 

4 

0 

3.08 

0.76 

0.58 

3 

(1.03) 

5f 

1 

0 

8 

3 

2 

0 

2.54 

0.75 

0.56 

2 

(0.33) 

Question  6 

6a 

1 

2 

6 

4 

1 

0 

2.31 

0.82 

0.67 

2 

(0.32) 

6b 

1 

1 

3 

7 

2 

0 

2.77 

0.80 

0.64 

3 

(105) 

6c 

2 

1 

9 

1 

1 

0 

2.17 

0.69 

0.47 

2 

(0.17) 

3d 

0 

4 

3 

4 

3 

0 

2.43 

1.12 

1.24 

3 

0.03 

Be 

4 

6 

2 

2 

0 

1.60 

0.99 

0.98 

2 

0.67 

6f 

4 

1 

6 

3 

0 

0 

2.20 

0.60 

0.36 

2 

(0.38) 

6g 

1 

1 

3 

6 

3 

0 

2.85 

0.86 

0.75 

3 

(0.94) 

Question  7 

7a 

2 

4 

6 

2 

0 

0 

1.83 

0.69 

0.47 

2 

(0.24) 

7b 

2 

4 

8 

0 

0 

0 

1.67 

0.47 

0.22 

2 

(0.97) 

7c 

2 

4 

8 

0 

0 

0 

1.67 

0.47 

0.22 

2 

(0.97) 

Question  8 

8a 

3 

1 

5 

3 

2 

0 

2.55 

0.89 

0.79 

2 

(0.21) 

8b 

4 

1 

3 

3 

3 

0 

2.80 

0.98 

0.96 

3 

(0.14) 

8c 

4 

2 

4 

1 

3 

0 

2.50 

1.12 

1.25 

2 

0.27 

8d 

5 

2 

4 

2 

1 

0 

2.22 

0.92 

0.84 

2 

0.40 

Question  9 

9 

5 

0 

7 

1 

1 

0 

2.33 

0.63 

0.40 

2 

0.13 

Question  1 0 

10a 

0 

0 

4 

4 

6 

0 

3.14 

0.83 

0.69 

3 

(0.31) 

10b 

0 

1 

6 

3 

4 

0 

2.71 

0.96 

0.92 

3 

0.12 

10c 

1 

0 

5 

6 

2 

0 

2.77 

0.70 

0.49 

3 

(0.99) 

10d 

1 

1 

6 

4 

2 

0 

2.54 

0.84 

0.71 

2 

(0.42) 
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Cost/Price  Analyst  Group  Data 


No  Response 

#1 

#2 

#3 

#4 

#5 

Mean 

Std  Dev 

Variation 

Median 

Skew 

Question  1 

Benchmarking 

26 

1 

13 

12 

6 

1 

2.79 

0.88 

0.77 

3 

0.20 

Validating 

19 

1 

13 

19 

7 

0 

2.80 

0.75 

0.56 

3 

(0.39) 

Program  Est 

19 

5 

22 

12 

1 

0 

2.23 

0.69 

0.47 

2 

(0.22) 

Budgeting 

14 

7 

23 

13 

2 

0 

2.22 

0.76 

0.57 

2 

(0.31) 

ICE 

20 

7 

13 

11 

8 

0 

2.51 

1.01 

1.02 

2 

0.09 

Software 

20 

6 

11 

15 

7 

0 

2.59 

0.95 

0.91 

3 

(0.05) 

Repair  Parts 

18 

2 

9 

21 

9 

0 

2.90 

0.79 

0.62 

3 

(0.49) 

R&D 

14 

5 

13 

13 

12 

2 

2.84 

1.07 

1.15 

3 

(0.24) 

Service 

16 

2 

4 

19 

18 

0 

3.23 

0.80 

0.64 

3 

(0.77) 

Hardware 

14 

5 

10 

24 

6 

0 

2.69 

0.84 

0.70 

3 

(0.60) 

Construction 

22 

4 

9 

14 

10 

0 

2.81 

0.95 

0.91 

3 

(0.06) 

Commodity 

25 

1 

5 

14 

14 

0 

3.21 

0.80 

0.63 

3 

(0.14) 

Question  2 

CE 

15 

10 

17 

10 

6 

1 

2.34 

1.03 

1.06 

2 

0.18 

PD&RR 

13 

8 

19 

10 

8 

1 

2.46 

1.04 

1.07 

2 

0.05 

E&MD 

11 

6 

19 

18 

5 

0 

2.46 

0.82 

0.68 

2 

(0.49) 

PFD&OS 

12 

7 

13 

20 

7 

0 

2.57 

0.92 

0.84 

3 

(0.45) 

Question  3 

FFP 

12 

9 

12 

18 

7 

1 

2.55 

1.03 

1.06 

3 

(0.20) 

FPE 

22 

5 

5 

19 

7 

1 

2.84 

0.97 

0.95 

3 

(0.05) 

FPIS 

23 

4 

10 

19 

3 

0 

2.58 

0.79 

0.63 

3 

(0.09) 

FPR(P) 

20 

5 

9 

21 

4 

0 

2.62 

0.84 

0.70 

3 

(0.24) 

FPR(R) 

21 

5 

7 

20 

6 

0 

2.71 

0.89 

0.78 

3 

(0.17) 

FPiF 

20 

5 

10 

18 

6 

0 

2.64 

0.89 

0.79 

3 

(0.16) 

CPIF 

15 

5 

10 

20 

9 

0 

2.75 

0.91 

0.82 

3 

(0.46) 

CPAF 

17 

6 

12 

16 

7 

1 

2.64 

1.00 

0.99 

3 

(0.11) 

CPFF 

16 

7 

13 

15 

7 

1 

2.58 

1.02 

1.03 

3 

(0.07) 

T&M 

18 

3 

4 

17 

13 

4 

3.27 

1.01 

1.03 

3 

(0.37) 

Labor  Hour 

20 

3 

6 

14 

12 

4 

3.21 

1.07 

1.14 

3 

(0.17) 

Question  4 

Pre-Award 

10 

11 

17 

15 

6 

0 

2.33 

0.96 

0.91 

2 

(0.15) 

Post-Award 

13 

8 

7 

17 

13 

1 

2.83 

1.09 

1.19 

3 

(0.37) 
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Question  5 

No  Response 

#1 

#2 

#3 

#4 

#5 

Mean 

Std  Dev 

Variation 

Median 

Skew 

5a 

9 

8 

15 

16 

11 

0 

2.60 

1.00 

1.00 

3 

(0.36) 

5b 

14 

2 

19 

18 

6 

0 

2.62 

0.77 

0.59 

3 

(0.53) 

5c 

11 

24 

13 

4 

0 

1.51 

0.86 

0.73 

2 

(0.03) 

5d 

8 

9 

28 

10 

3 

1 

2.20 

0.86 

0.75 

2 

0.25 

5e 

18 

1 

11 

16 

13 

0 

3.00 

0.83 

0.68 

3 

(0.45) 

5f 

15 

6 

19 

14 

5 

0 

2.41 

0.86 

0.74 

2 

(0.22) 

Question  6 

6a 

16 

4 

24 

12 

3 

0 

2.33 

0.74 

0.55 

2 

(0.28) 

6b 

12 

2 

14 

20 

11 

0 

2.85 

0.82 

0.68 

3 

(0.72) 

6c 

14 

5 

25 

12 

3 

0 

2.29 

0.75 

0.56 

2 

(0.31) 

6d 

9 

4 

19 

22 

5 

0 

2.56 

0.78 

0.61 

3 

(0.70) 

Be 

11 

7 

16 

20 

5 

0 

2.48 

0.87 

0.75 

3 

(0.48) 

6f 

13 

6 

26 

10 

4 

0 

2.26 

0.79 

0.63 

2 

(0.19) 

6g 

5 

20 

13 

5 

0 

0 

1.61 

0.84 

0.71 

2 

(0.19) 

Question  7 

7a 

8 

20 

27 

2 

2 

0 

1.73 

0.72 

0.51 

2 

0.48 

7b 

8 

18 

29 

4 

0 

0 

1.73 

0.60 

0.36 

2 

(0.36) 

7c 

9 

18 

31 

1 

0 

0 

1.66 

0.51 

0.26 

2 

(0.73) 

Question  8 

8a 

9 

6 

29 

7 

8 

0 

2.34 

0.89 

0.78 

2 

0.01 

8b 

9 

6 

34 

7 

3 

0 

2.14 

0.69 

0.48 

2 

(0.19) 

8c 

8 

16 

27 

5 

3 

0 

1.90 

0.80 

0.64 

2 

0.35 

8d 

15 

9 

25 

6 

4 

0 

2.11 

0.83 

0.69 

2 

0.10 

Question  9 

9 

16 

4 

17 

14 

7 

1 

2.63 

0.94 

0.88 

3 

(0.15) 

Question  10 

10a 

11 

3 

19 

22 

4 

0 

2.56 

0.73 

0.54 

3 

(0.75) 

10b 

20 

0 

11 

22 

6 

0 

2.87 

0.65 

0.42 

3 

(0.46) 

10c 

11 

3 

20 

20 

5 

0 

2.56 

0.76 

0.58 

3 

(0.65) 

10d 

12 

10 

12 

19 

6 

0 

2.45 

0.96 

0.93 

3 

(0.30) 
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Contract  Negotiator  Group  Data 


No  Response 

#1 

#2 

#3 

#4 

#5 

Mean 

Std  Dev 

Variation 

Median 

Skew 

Question  1 

Benchmarking 

5 

0 

1 

2 

0 

0 

2.67 

0.47 

0.22 

3 

0.81 

Validating 

4 

0 

2 

2 

0 

0 

2.50 

0.50 

0.25 

3 

0.27 

Program  Est 

4 

2 

2 

0 

0 

0 

1.50 

0.50 

0.25 

2 

0.62 

Budgeting 

4 

3 

1 

0 

0 

0 

1.25 

0.43 

0.19 

1 

0.82 

ICE 

4 

1 

0 

3 

0 

0 

2.50 

0.87 

0.75 

3 

0.48 

Software 

3 

2 

0 

3 

0 

0 

2.20 

0.98 

0.96 

3 

0.34 

Repair  Parts 

4 

1 

1 

2 

0 

0 

2.25 

0.83 

0.69 

3 

0.62 

R&D 

2 

2 

1 

1 

2 

0 

2.50 

1.26 

1.58 

3 

0.26 

Service 

2 

1 

3 

1 

1 

0 

2.33 

0.94 

0.89 

2 

0.16 

Hardware 

3 

2 

2 

1 

0 

0 

1.80 

0.75 

0.56 

2 

0.49 

Construction 

4 

1 

3 

0 

0 

0 

1.75 

0.43 

0.19 

2 

0.31 

Commodity 

5 

0 

0 

1 

2 

0 

3.67 

0.47 

0.22 

4 

0.74 

Question  2 

CE 

2 

1 

2 

2 

1 

0 

2.50 

0.96 

0.92 

3 

(0.09) 

PD&RR 

2 

2 

1 

2 

1 

0 

2.33 

1.11 

1.22 

3 

0.22 

E&MD 

1 

2 

2 

2 

1 

0 

2.29 

1.03 

1.06 

2 

0.00 

PFD&OS 

2 

2 

1 

1 

2 

0 

2.50 

1.26 

1.58 

3 

0.26 

Question  3 

FFP 

2 

1 

1 

4 

0 

0 

2.50 

0.76 

0.58 

3 

(0.62) 

FPE 

4 

1 

1 

2 

0 

0 

2.25 

0.83 

0.69 

3 

0.62 

FPIS 

4 

1 

1 

2 

0 

0 

2.25 

0.83 

0.69 

3 

0.62 

FPR(P) 

4 

1 

1 

2 

0 

0 

2.25 

0.83 

0.69 

3 

0.62 

FPR(R) 

4 

1 

1 

2 

0 

0 

2.25 

0.83 

0.69 

3 

0.62 

FPIF 

4 

1 

1 

2 

0 

0 

2.25 

0.83 

0.69 

3 

0.62 

CPIF 

4 

2 

2 

0 

0 

0 

1.50 

0.50 

0.25 

2 

0.62 

CPAF 

3 

2 

2 

1 

0 

0 

1.80 

0.75 

0.56 

2 

0.49 

CPFF 

3 

2 

2 

0 

1 

0 

2.00 

1.10 

1.20 

2 

1.12 

T&M 

3 

1 

2 

2 

0 

0 

2.20 

0.75 

0.56 

2 

0.11 

Labor  Hour 

4 

1 

2 

0 

1 

0 

2.25 

1.09 

1.19 

2 

1.19 

Question  4 

Pre-Award 

1 

1 

2 

2 

2 

0 

2.71 

1.03 

1.06 

3 

(0.48) 

Post-Award 

2 

1 

2 

1 

1 

1 

2.83 

1.34 

1.81 

3 

0.34 
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Question  5 

No  Response 

#1 

#2 

#3 

#4 

#5 

Mean 

Std  Dev 

Variation 

Median 

Skew 

5a 

2 

0 

3 

3 

0 

0 

2.50 

0.50 

0.25 

3 

(0.88) 

5b 

2 

1 

4 

0 

1 

0 

2.17 

0.90 

0.81 

2 

0.41 

5c 

1 

3 

2 

2 

0 

0 

1.86 

0.83 

0.69 

2 

0.04 

5d 

1 

1 

4 

2 

0 

0 

2.14 

0.64 

0.41 

2 

(0.86) 

5e 

3 

1 

2 

1 

1 

0 

2.40 

1.02 

1.04 

2 

0.50 

5f 

2 

0 

6 

0 

0 

0 

2.00 

0.00 

0.00 

2 

(1.44) 

Question  6 

6a 

1 

1 

5 

1 

0 

0 

2.00 

0.53 

0.29 

2 

(1-03) 

6b 

2 

1 

2 

2 

1 

0 

2.50 

0.96 

0.92 

3 

(0.09) 

6c 

2 

1 

2 

2 

1 

0 

2.50 

0.96 

0.92 

3 

(0.09) 

6d 

2 

2 

4 

0 

0 

0 

1.67 

0.47 

0.22 

2 

(0.62) 

Be 

2 

2 

4 

0 

0 

0 

1.67 

0.47 

0.22 

2 

(0.62) 

6f 

2 

3 

0 

3 

0 

0 

2.00 

1.00 

1.00 

2 

0.25 

6g 

3 

1 

2 

2 

0 

0 

2.20 

0.75 

0.56 

2 

0.11 

Question  7 

7a 

2 

2 

3 

0 

1 

0 

2.00 

1.00 

1.00 

2 

0.76 

7b 

2 

2 

3 

1 

0 

0 

1.83 

0.64 

0.41 

2 

(0.27) 

7c 

2 

2 

3 

1 

0 

0 

1.83 

0.69 

0.47 

2 

(0.04) 

Question  8 

8a 

2 

1 

2 

2 

1 

0 

2.50 

0.96 

0.92 

3 

(0.09) 

8b 

2 

1 

2 

2 

1 

0 

2.50 

0.96 

0.92 

3 

(0.09) 

8c 

2 

0 

4 

1 

1 

0 

2.50 

0.76 

0.58 

2 

(0.16) 

8d 

2 

0 

4 

1 

1 

0 

2.50 

0.76 

0.58 

2 

(0.16) 

Question  9 

9 

2 

0 

3 

2 

1 

0 

2.67 

0.75 

0.56 

3 

(0.40) 

Question  10 

10a 

1 

0 

5 

1 

1 

0 

2.43 

0.73 

0.53 

2 

(0.31) 

10b 

1 

0 

3 

2 

2 

0 

2.86 

0.83 

0.69 

3 

(0.76) 

10c 

2 

1 

2 

3 

0 

0 

2.33 

0.75 

0.56 

3 

(0.47) 

10d 

2 

1 

3 

1 

1 

0 

2.33 

0.94 

0.89 

2 

0.16 
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Contracting  Officer  Group  Data 


No  Response 

#1 

#2 

#3^ 

#4 

#5 

Mean 

Std  Dev 

Variation 

Median 

Skew 

Question  1 

Benchmarking 

18 

1 

11 

16 

3 

0 

2.68 

0.69 

0.48 

3 

(0.11) 

Validating 

13 

2 

11 

21 

2 

0 

2.64 

0.67 

0.45 

3 

(0.52) 

Program  Est 

12 

3 

18 

15 

1 

0 

2.38 

0.67 

0.45 

2 

(0.44) 

Budgeting 

13 

5 

18 

11 

2 

0 

2.28 

0.77 

0.59 

2 

(0.13) 

ICE 

22 

1 

14 

10 

2 

0 

2.48 

0.69 

0.47 

2 

0.24 

Software 

22 

4 

10 

8 

5 

0 

2.52 

0.96 

0.92 

2 

0.46 

Repair  Parts 

16 

0 

8 

22 

3 

0 

2.85 

0.56 

0.31 

3 

(0.43) 

R&D 

17 

3 

11 

11 

6 

1 

2.72 

0.98 

0.95 

3 

0.14 

Service 

16 

1 

8 

11 

12 

1 

3.12 

0.91 

0.83 

3 

(0.17) 

Hardware 

9 

0 

17 

20 

3 

0 

2.65 

0.61 

0.38 

3 

(0.81) 

Construction 

25 

0 

10 

9 

5 

0 

2.79 

0.76 

0.58 

3 

0.44 

Commodity 

26 

0 

3 

12 

8 

0 

3.22 

0.66 

0.43 

3 

0.34 

Question  2 

CE 

12 

8 

6 

15 

7 

1 

2.65 

1.10 

1.20 

3 

(0.04) 

PD&RR 

12 

3 

11 

21 

2 

0 

2.59 

0.72 

0.51 

3 

(0.53) 

E&MD 

12 

2 

16 

17 

2 

0 

2.51 

0.68 

0.47 

3 

(0.47) 

PFD&OS 

11 

1 

15 

17 

5 

0 

2.68 

0.73 

0.53 

3 

(0.51) 

Question  3 

FFP 

9 

4 

15 

15 

6 

0 

2.58 

0.86 

0.74 

3 

(0.39) 

FPE 

14 

1 

13 

14 

7 

0 

2.77 

0.80 

0.63 

3 

(0.27) 

FPIS 

16 

0 

13 

16 

4 

0 

2.73 

0.66 

0.44 

3 

(0.26) 

FPR(P) 

15 

0 

13 

17 

4 

0 

2.74 

0.66 

0.43 

3 

(0.35) 

FPR(R) 

16 

0 

13 

15 

5 

0 

2.76 

0.70 

0.49 

3 

(0.23) 

FPIF 

13 

0 

17 

15 

4 

0 

2.64 

0.67 

0.45 

3 

(0.40) 

CPIF 

15 

2 

11 

16 

5 

0 

2.71 

0.79 

0.62 

3 

(0.23) 

CPAF 

13 

2 

11 

18 

5 

0 

2.72 

0.77 

0.59 

3 

(0.39) 

CPFF 

13 

3 

9 

20 

4 

0 

2.69 

0.78 

0.60 

3 

(0.41) 

T&M 

15 

0 

8 

14 

11 

1 

3.15 

0.81 

0.65 

3 

(0.31) 

Labor  Hour 

14 

0 

10 

12 

12 

1 

3.11 

0.85 

0.73 

3 

(0.32) 

Question  4 

Pre-Award 

9 

3 

15 

19 

3 

0 

2.55 

0.74 

0.55 

3 

(0.61) 

Post-Award 

11 

0 

10 

14 

14 

0 

3.11 

0.79 

0.62 

3 

(0.63) 
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Question  5 

No  Response 

#1 

#2 

#3 

#4 

#5 

Mean 

Std  Dev 

Variation 

Median 

Skew 

5a 

10 

3 

17 

15 

4 

0 

2.51 

0.78 

0.61 

2 

(0.41) 

5b 

10 

2 

20 

14 

3 

0 

2.46 

0.71 

0.50 

2 

(0.46) 

5c 

8 

7 

24 

9 

1 

0 

2.10 

0.69 

0.48 

2 

(0.36) 

5d 

11 

1 

29 

5 

3 

0 

2.26 

0.64 

0.40 

2 

(0.25) 

5e 

11 

0 

5 

23 

10 

0 

3.13 

0.61 

0.38 

3 

(0.86) 

5f 

9 

0 

26 

13 

1 

0 

2.38 

0.53 

0.28 

2 

(0.78) 

Question  6 

6a 

11 

6 

22 

6 

4 

0 

2.21 

0.83 

0.69 

2 

0.04 

6b 

9 

3 

12 

21 

4 

0 

2.65 

0.76 

0.58 

3 

(0.66) 

6c 

9 

4 

22 

10 

4 

0 

2.35 

0.79 

0.63 

2 

(0.24) 

6d 

11 

1 

21 

12 

4 

0 

2.50 

0.72 

0.51 

2 

(0.37) 

6e 

10 

4 

19 

14 

2 

0 

2.36 

0.73 

0.54 

2 

(0.41) 

6f 

10 

6 

29 

2 

2 

0 

2.00 

0.64 

0.41 

2 

(0.13) 

6g 

11 

2 

18 

14 

4 

0 

2.53 

0.75 

0.57 

2 

(0.38) 

Question  7 

7a 

8 

10 

28 

3 

0 

0 

1.83 

0.54 

0.29 

2 

(0.64) 

7b 

8 

12 

27 

2 

0 

0 

1.76 

0.53 

0.28 

2 

(0.61) 

7c 

8 

14 

26 

1 

0 

0 

1.68 

0.52 

0.27 

2 

(0.61) 

Question  8 

8a 

10 

7 

24 

7 

1 

0 

2.05 

0.68 

0.46 

2 

(0.25) 

8b 

10 

7 

25 

7 

0 

0 

2.00 

0.60 

0.36 

2 

(0.48) 

8c 

9 

11 

25 

4 

0 

0 

1.83 

0.59 

0.34 

2 

(0-43) 

8d 

12 

6 

27 

4 

0 

0 

1.95 

0.52 

0.27 

2 

(0.50) 

Question  9 

16 

3 

23 

5 

2 

0 

2.18 

0.67 

0.45 

2 

0.04 

9 

Question  1 0 

10a 

11 

2 

16 

17 

3 

0 

2.55 

0.71 

0.51 

3 

(0.49) 

10b 

14 

0 

8 

22 

5 

0 

2.91 

0.60 

0.36 

3 

(0.55) 

10c 

12 

1 

7 

22 

7 

0 

2.95 

0.70 

0.48 

3 

(0.64) 

10d 

11 

4 

19 

13 

2 

0 

2.34 

0.74 

0.54 

2 

(0.33) 
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Program  Staff  Group  Data 


No  Response 

#1 

#2 

#3 

#4 

#5 

Mean 

Std  Dev 

Variation 

Median 

Skew 

Question  1 

Benchmarking 

3 

0 

0 

1 

0 

0 

3.00 

0.00 

0.00 

3 

2.00 

Validating 

1 

0 

0 

3 

0 

0 

3.00 

0.00 

0.00 

3 

(2.00) 

Program  Est 

1 

0 

0 

3 

0 

0 

3.00 

0.00 

0.00 

3 

(2.00) 

Budgeting 

1 

0 

0 

3 

0 

0 

3.00 

0.00 

0.00 

3 

(2.00) 

ICE 

2 

0 

1 

1 

0 

0 

2.50 

0.50 

0.25 

3 

0.37 

Software 

1 

0 

1 

1 

0 

3.00 

0.82 

0.67 

3 

(0.75) 

Repair  Parts 

2 

0 

0 

1 

0 

3.50 

0.50 

0.25 

4 

0.20 

R&D 

1 

0 

0 

2 

0 

3.33 

0.47 

0.22 

3 

(1.54) 

Service 

2 

0 

0 

1 

0 

3.50 

0.50 

0.25 

4 

0.20 

Hardware 

1 

0 

0 

2 

0 

3.33 

0.47 

0.22 

3 

(1.54) 

Construction 

3 

0 

0 

1 

0 

0 

3.00 

0.00 

0.00 

3 

2.00 

Commodity 

3 

0 

0 

1 

0 

0 

3.00 

0.00 

0.00 

3 

2.00 

Question  2 

CE 

2 

0 

2 

0 

0 

0 

2.00 

0.00 

0.00 

2 

0.00 

PD&RR 

2 

0 

2 

0 

0 

0 

2.00 

0.00 

0.00 

2 

0.00 

E&MD 

2 

0 

0 

2 

0 

0 

3.00 

0.00 

0.00 

3 

0.00 

PFD&OS 

0 

0 

0 

1 

3 

0 

3.75 

0.43 

0.19 

4 

(2.00) 

Question  3 

FFP 

2 

0 

1 

0 

0 

2.50 

0.50 

0.25 

3 

0.37 

FPE 

3 

0 

0 

0 

0 

3.00 

0.00 

0.00 

3 

2.00 

FPIS 

3 

0 

0 

0 

0 

3.00 

0.00 

0.00 

3 

2.00 

FPR(P) 

3 

0 

0 

0 

0 

3.00 

0.00 

0.00 

3 

2.00 

FPR(R) 

3 

0 

0 

0 

0 

3.00 

0.00 

0.00 

3 

2.00 

FPIF 

2 

0 

1 

0 

0 

2.50 

0.50 

0.25 

3 

0.37 

CPIF 

2 

0 

1 

0 

1 

0 

3.00 

1.00 

1.00 

3 

0.85 

CPAF 

2 

0 

1 

0 

1 

0 

3.00 

1.00 

1.00 

3 

0.85 

CPFF 

2 

0 

1 

0 

1 

0 

3.00 

1.00 

1.00 

3 

0.85 

T&M 

2 

0 

0 

0 

2 

0 

4.00 

0.00 

0.00 

4 

0.00 

Labor  Hour 

2 

0 

0 

0 

2 

0 

4.00 

0.00 

0.00 

4 

0.00 

Question  4 

Pre-Award 

1 

0 

0 

3 

0 

0 

3.00 

0.00 

0.00 

3 

(2.00) 

Post-Award 

1 

0 

0 

1 

2 

0 

3.67 

0.47 

0.22 

4 

(1.66) 
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Question  5 

No  Response 

#1 

#2 

#3 

#4 

#5 

Mean 

Std  Dev 

Variation 

Median 

Skew 

5a 

0 

0 

0 

1 

3 

0 

3.75 

0.43 

0.19 

4 

(2.00) 

5b 

1 

0 

0 

2 

1 

0 

3.33 

0.47 

0.22 

3 

(1.54) 

5c 

1 

0 

2 

0 

1 

0 

2.67 

0.94 

0.89 

2 

0.00 

5d 

0 

0 

2 

1 

1 

0 

2.75 

0.83 

0.69 

3 

0.85 

5e 

0 

0 

2 

0 

2 

0 

3.00 

1.00 

1.00 

3 

0.00 

5f 

0 

0 

3 

1 

0 

0 

2.25 

0.43 

0.19 

2 

2.00 

Question  6 

6a 

1 

0 

1 

2 

0 

0 

2.67 

0.47 

0.22 

3 

(141) 

6b 

0 

0 

2 

2 

0 

0 

2.50 

0.50 

0.25 

3 

0.00 

6c 

0 

1 

1 

2 

0 

0 

2.25 

0.83 

0.69 

3 

(0.85) 

5d 

0 

0 

3 

1 

0 

0 

2.25 

0.43 

0.19 

2 

2.00 

5e 

0 

0 

2 

2 

0 

0 

2.50 

0.50 

0.25 

3 

0.00 

6f 

0 

2 

2 

0 

0 

0 

1.50 

0.50 

0.25 

2 

0.00 

5g 

1 

0 

1 

2 

0 

0 

2.67 

0.47 

0.22 

3 

(1.41) 

Question  7 

7a 

0 

0 

2 

2 

0 

0 

2.50 

0.50 

0.25 

3 

0.00 

7b 

0 

1 

2 

1 

0 

0 

2.00 

0.71 

0.50 

2 

0.00 

7c 

0 

0 

4 

0 

0 

0 

2.00 

0.00 

0.00 

2 

0.00 

Question  8 

8a 

0 

0 

4 

0 

0 

0 

2.00 

0.00 

0.00 

2 

0.00 

8b 

0 

0 

4 

0 

0 

0 

2.00 

0.00 

0.00 

2 

0.00 

8c 

0 

0 

4 

0 

0 

0 

2.00 

0.00 

0.00 

2 

0.00 

8d 

0 

0 

4 

0 

0 

0 

2.00 

0.00 

0.00 

2 

0.00 

Question  9 

9 

2 

0 

1 

1 

0 

0 

2.50 

0.50 

0.25 

3 

0.37 

Question  1 0 

10a 

1 

0 

0 

3 

0 

0 

3.00 

0.00 

0.00 

3 

(2.00) 

10b 

2 

0 

1 

1 

0 

0 

2.50 

0.50 

0.25 

3 

0.37 

10c 

0 

0 

3 

1 

0 

0 

2.25 

0.43 

0.19 

2 

2.00 

10d 

2 

0 

0 

2 

0 

0 

3.00 

0.00 

0.00 

3 

0.00 
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Program  Manager  Group  Data 


No  Response 

#1 

#2 

#3 

#4 

#5 

Mean 

Std  Dev 

Variation 

Median 

Skew 

Question  1 

Benchmarking 

1 

4 

1 

1 

0 

0 

1.50 

0.76 

0.58 

1 

0.86 

Validating 

1 

3 

1 

1 

1 

0 

2.00 

1.15 

1.33 

2 

0.71 

Program  Est 

0 

6 

0 

1 

0 

0 

1.29 

0.70 

0.49 

1 

2.65 

Budgeting 

0 

4 

2 

1 

0 

0 

1.57 

0.73 

0.53 

1 

1.11 

ICE 

1 

3 

0 

3 

0 

0 

2.00 

1.00 

1.00 

2 

0.03 

Software 

1 

1 

3 

2 

0 

0 

2.17 

0.69 

0.47 

2 

(0.77) 

Repair  Parts 

2 

1 

2 

1 

1 

0 

2.40 

1.02 

1.04 

2 

0.26 

R&D 

2 

0 

3 

1 

1 

0 

2.60 

0.80 

0.64 

2 

(0.11) 

Service 

1 

0 

0 

32 

0 

0 

3.00 

0.69 

0.47 

3 

(1.42) 

Hardware 

2 

2 

1 

3 

0 

0 

2.17 

0.89 

0.80 

2 

0.22 

Construction 

0 

2 

1 

4 

0 

0 

2.29 

0.88 

0.78 

3 

(0.76) 

Commodity 

2 

0 

0 

4 

1 

0 

3.20 

0.40 

0.16 

3 

(0.98) 

Question  2 

CE 

0 

4 

1 

1 

1 

0 

1.86 

1.12 

1.27 

1 

1.15 

PD&RR 

0 

4 

1 

1 

1 

0 

1.86 

1.03 

1.06 

1 

1.78 

E&MD 

0 

2 

2 

2 

1 

0 

2.29 

1.03 

1.06 

2 

0.25 

PFD&OS 

1 

2 

4 

0 

0 

0 

1.67 

0.90 

0.81 

3 

(0.60) 

Question  3 

FFP 

1 

2 

3 

0 

0 

2.33 

0.75 

0.56 

3 

(0.91) 

FPE 

3 

1 

2 

0 

0 

2.25 

0.83 

0.69 

3 

0.36 

FPIS 

2 

2 

2 

0 

0 

2.20 

0.75 

0.56 

2 

(0.22) 

FPR(P) 

2 

2 

2 

0 

0 

2.20 

0.75 

0.56 

2 

(0.22) 

FPR(R) 

2 

3 

0 

0 

2.00 

0.63 

0.40 

2 

(0.24) 

FPIF 

2 

3 

0 

0 

2.00 

0.63 

0.40 

2 

(0.24) 

CPIF 

2 

2 

2 

0 

0 

1.80 

0.75 

0.56 

2 

0.25 

CPAF 

2 

2 

2 

0 

0 

1.80 

0.75 

0.56 

2 

0.25 

CPFF 

2 

2 

2 

0 

0 

1.80 

0.75 

0.56 

2 

0.25 

T&M 

1 

2 

1 

2 

1 

0 

2.33 

1.11 

1.22 

3 

0.00 

Labor  Hour 

2 

0 

1 

3 

1 

0 

3.00 

0.63 

0.40 

3 

(0.68) 

Question  4 

Pre-Award 

0 

4 

2 

0 

0 

0 

1.33 

0.73 

0.53 

1 

1.11 

Post-Award 

0 

2 

2 

2 

0 

0 

2.00 

0.83 

0.69 

2 

(0.35) 
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Question  5 

No  Response 

#1 

#2 

#3 

#4 

#5 

Mean 

Std  Dev 

Variation 

Median 

Skew 

5a 

0 

3 

2 

1 

1 

0 

2.00 

1.07 

1.14 

2 

0.91 

5b 

0 

3 

2 

0 

2 

0 

2.14 

1.25 

1.55 

2 

0.80 

5c 

0 

3 

2 

1 

1 

0 

2.00 

1.07 

1.14 

2 

0.91 

5d 

0 

4 

1 

2 

0 

0 

1.71 

0.88 

0.78 

1 

0.76 

5e 

0 

3 

3 

0 

1 

0 

1.86 

0.99 

0.98 

2 

1.52 

5f 

1 

3 

2 

0 

1 

0 

1.83 

1.07 

1.14 

2 

1.14 

Question  6 

6a 

0 

4 

2 

0 

1 

0 

1.71 

1.03 

1.06 

1 

1.78 

6b 

0 

3 

1 

2 

1 

0 

2.14 

1.12 

1.27 

2 

0.41 

6c 

1 

1 

2 

2 

1 

0 

2.50 

0.96 

0.92 

3 

(0.35) 

6d 

0 

2 

4 

1 

0 

0 

1.86 

0.64 

0.41 

2 

0.17 

Be 

0 

2 

4 

1 

0 

0 

1.86 

0.64 

0.41 

2 

0.17 

6f 

0 

1 

6 

0 

0 

0 

1.86 

0.35 

0.12 

2 

(2.65) 

6g 

1 

0 

3 

1 

2 

0 

2.83 

0.90 

0.81 

3 

(0.57) 

Question  7 

7a 

1 

3 

0 

0 

3 

0 

2.50 

1.50 

2.25 

3 

0.21 

7b 

0 

3 

2 

0 

2 

0 

2.14 

1.25 

1.55 

2 

0.80 

7c 

0 

4 

1 

0 

2 

0 

2.00 

1.31 

1.71 

1 

0.99 

Question  8 

8a 

2 

1 

2 

1 

1 

0 

2.40 

1.02 

1.04 

2 

0.26 

8b 

1 

1 

2 

1 

2 

0 

2.67 

1.11 

1.22 

3 

(0.26) 

8c 

1 

1 

5 

0 

0 

0 

1.83 

0.37 

0.14 

2 

(1.76) 

8d 

1 

0 

6 

0 

0 

0 

2.00 

0.00 

0.00 

2 

(2.65) 

Question  9 

9 

1 

2 

4 

0 

0 

0 

1.67 

0.47 

0.22 

2 

(1.11) 

Question  10 

10a 

1 

1 

2 

1 

2 

0 

2.67 

1.11 

1.22 

3 

(0.26) 

10b 

1 

0 

2 

4 

0 

0 

2.67 

0.47 

0.22 

3 

(1.78) 

10c 

1 

1 

1 

3 

1 

0 

2.67 

0.94 

0.89 

3 

(0.71) 

10d 

1 

2 

2 

1 

1 

0 

2.17 

1.07 

1.14 

2 

0.35 
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Engineer  Group  Data 


No  Response 

#1 

#2 

#3 

#4 

#5 

Mean 

Std  Dev 

Variation 

Median 

Skew 

Question  1 

Benchmarking 

4 

1 

2 

2 

3 

0 

2.88 

1.05 

1.11 

3 

0.02 

Validating 

3 

6 

1 

2 

0 

0 

1.56 

0.83 

0.69 

1 

0.81 

Program  Est 

3 

4 

3 

2 

0 

0 

1.78 

0.79 

0.62 

2 

0.26 

Budgeting 

1 

7 

3 

1 

0 

0 

1.45 

0.66 

0.43 

1 

0.67 

ICE 

3 

6 

2 

1 

0 

0 

1.44 

0.68 

0.47 

1 

0.71 

Software 

2 

6 

1 

2 

1 

0 

1.80 

1.08 

1.16 

1 

0.85 

Repair  Parts 

3 

2 

3 

4 

0 

0 

2.22 

0.79 

0.62 

2 

(0.29) 

R&D 

3 

4 

3 

2 

0 

0 

1.78 

0.79 

0.62 

2 

0.26 

Service 

2 

2 

0 

5 

3 

0 

2.90 

1.04 

1.09 

3 

(0.67) 

Hardware 

2 

4 

1 

5 

0 

0 

2.10 

0.94 

0.89 

3 

(0.16) 

Construction 

4 

2 

3 

3 

0 

0 

2.13 

0.78 

0.61 

2 

0.06 

Commodity 

5 

1 

1 

3 

2 

0 

2.86 

0.99 

0.98 

3 

0.21 

Question  2 

CE 

1 

5 

3 

2 

1 

0 

1.91 

1.00 

0.99 

2 

0.58 

PD&RR 

1 

4 

5 

2 

0 

0 

1.82 

0.72 

0.51 

2 

(0.14) 

E&MD 

1 

5 

1 

5 

0 

0 

2.00 

0.95 

0.91 

2 

(0.09) 

PFD&OS 

1 

3 

3 

3 

2 

0 

2.36 

1.07 

1.14 

2 

(0.05) 

Question  3 

FFP 

3 

3 

4 

2 

0 

0 

1.89 

0.74 

0.54 

2 

(0.00) 

FPE 

7 

3 

1 

0 

0 

1.60 

0.80 

0.64 

1 

1.50 

FPIS 

7 

3 

1 

0 

0 

1.60 

0.80 

0.64 

1 

1.50 

FPR(P) 

6 

3 

2 

0 

0 

1.67 

0.75 

0.56 

2 

0.99 

FPR(R) 

6 

3 

2 

0 

0 

1.67 

0.75 

0.56 

2 

0.99 

FPIF 

4 

5 

2 

0 

0 

1.50 

0.71 

0.50 

1 

0.76 

CPIF 

4 

3 

5 

0 

0 

0 

1.63 

0.48 

0.23 

2 

(0.18) 

CPAF 

5 

3 

4 

0 

0 

0 

1.57 

0.49 

0.24 

2 

0.18 

CPFF 

4 

3 

3 

2 

0 

0 

1.88 

0.78 

0.61 

2 

0.31 

T&M 

4 

1 

4 

1 

2 

0 

2.50 

1.00 

1.00 

2 

0.30 

Labor  Hour 

4 

1 

4 

1 

2 

0 

2.50 

1.00 

1.00 

2 

0.30 

Question  4 

Pre-Award 

2 

4 

3 

3 

0 

0 

1.90 

0.83 

0.69 

2 

0.00 

Post-Award 

2 

2 

3 

4 

1 

0 

2.40 

0.92 

0.84 

3 

(0.31) 
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Question  5 

No  Response 

#1 

#2 

#3 

#4 

#5 

Mean 

Std  Dev 

Variation 

Median 

Skew 

5a 

1 

2 

7 

2 

0 

0 

2.00 

0.60 

0.36 

2 

(0.77) 

5b 

3 

3 

3 

3 

0 

0 

2.00 

0.82 

0.67 

2 

0.00 

5c 

1 

6 

5 

0 

0 

0 

1.45 

0.50 

0.25 

1 

(0.44) 

5d 

0 

5 

6 

0 

0 

1 

1.83 

1.07 

1.14 

2 

2.28 

5e 

3 

1 

5 

3 

0 

0 

2.22 

0.63 

0.40 

2 

(0.49) 

5f 

3 

1 

7 

1 

0 

0 

2.00 

0.47 

0.22 

2 

(0.65) 

Question  6 

6a 

1 

2 

5 

3 

1 

0 

2.27 

0.86 

0.74 

2 

(0.19) 

6b 

0 

1 

2 

5 

4 

0 

3.00 

0.91 

0.83 

3 

(0.76) 

6c 

3 

3 

5 

1 

0 

0 

1.78 

0.63 

0.40 

2 

(0.13) 

6d 

2 

2 

4 

3 

1 

0 

2.30 

0.90 

0.81 

2 

(0.16) 

Be 

3 

1 

4 

3 

1 

0 

2.44 

0.83 

0.69 

2 

(0.19) 

6f 

0 

1 

8 

2 

1 

0 

2.25 

0.72 

0.52 

2 

1.05 

6g 

3 

2 

2 

4 

1 

0 

2.44 

0.96 

0.91 

3 

(0.12) 

Question  7 

7a 

1 

3 

6 

2 

0 

0 

1.91 

0.67 

0.45 

2 

(0.44) 

7b 

0 

3 

6 

3 

0 

0 

2.00 

0.71 

0.50 

2 

0.00 

7c 

0 

4 

7 

1 

0 

0 

1.75 

0.60 

0.35 

2 

0.17 

Question  8 

8a 

1 

2 

3 

5 

1 

0 

2.45 

0.89 

0.79 

3 

(0.58) 

8b 

2 

1 

4 

4 

1 

0 

2.50 

0.81 

0.65 

3 

(0.53) 

8c 

1 

3 

5 

3 

0 

0 

2.00 

0.74 

0.55 

2 

(0.41) 

8d 

5 

0 

4 

1 

2 

0 

2.71 

0.88 

0.78 

2 

0.33 

Question  9 

9 

3 

1 

5 

3 

0 

0 

2.22 

0.63 

0.40 

2 

(0.49) 

Question  1 0 

10a 

0 

0 

7 

3 

1 

1 

2.67 

0.94 

0.89 

2 

1.50 

10b 

5 

0 

3 

3 

1 

0 

2.71 

0.70 

0.49 

3 

0.10 

10c 

1 

1 

5 

4 

1 

0 

2.45 

0.78 

0.61 

2 

(0.59) 

10d 

1 

0 

7 

3 

1 

0 

2.45 

0.66 

0.43 

2 

(0.59) 
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Technical  Support  Group  Data 


No  Response 

#1 

#2 

#3 

#4 

#5 

Mean 

Std  Dev 

Variation 

Median 

Skew 

Question  1 

Benchmarking 

1 

1 

1 

2 

0 

0 

2.25 

0.83 

0.69 

3 

(0.54) 

Validating 

1 

0 

2 

2 

0 

0 

2.50 

0.50 

0.25 

3 

(1.36) 

Program  Est 

0 

1 

2 

1 

1 

0 

2.40 

1.02 

1.04 

2 

0.40 

Budgeting 

0 

0 

2 

3 

0 

0 

2.60 

0.49 

0.24 

3 

(0.61) 

ICE 

4 

1 

0 

0 

0 

0 

1.00 

0.00 

0.00 

1 

2.24 

Software 

1 

2 

1 

0 

0 

1 

2.25 

1.64 

2.69 

2 

1.52 

Repair  Parts 

0 

0 

0 

3 

1 

0 

3.25 

0.49 

0.24 

3 

0.61 

R&D 

2 

0 

0 

1 

1 

1 

4.00 

0.82 

0.67 

4 

(0.20) 

Service 

1 

0 

1 

2 

1 

0 

3.00 

0.71 

0.50 

3 

(1.12) 

Hardware 

1 

0 

1 

2 

1 

0 

3.00 

0.71 

0.50 

3 

(1.12) 

Construction 

2 

0 

0 

1 

2 

0 

3.67 

0.47 

0.22 

4 

(0.44) 

Commodity 

3 

0 

0 

0 

2 

0 

4.00 

0.00 

0.00 

4 

0.61 

Question  2 

CE 

1 

1 

2 

0 

1 

0 

2.25 

1.09 

1.19 

2 

0.55 

PD&RR 

0 

1 

1 

2 

0 

0 

2.25 

0.83 

0.69 

3 

(0.54) 

E&MD 

1 

0 

3 

1 

0 

0 

2.25 

0.43 

0.19 

2 

(1.29) 

PFD&OS 

1 

0 

1 

2 

1 

0 

3.00 

0.71 

0.50 

3 

(1.12) 

Question  3 

FFP 

1 

0 

0 

2 

2 

0 

3.50 

0.50 

0.25 

4 

(1.74) 

FPE 

2 

0 

0 

1 

2 

0 

3.67 

0.47 

0.22 

4 

(0.44) 

FPIS 

2 

0 

0 

2 

1 

0 

3.33 

0.47 

0.22 

3 

(0.38) 

FPR(P) 

2 

0 

0 

2 

1 

0 

3.33 

0.47 

0.22 

3 

(0.38) 

FPR(R) 

2 

0 

0 

2 

1 

0 

3.33 

0.47 

0.22 

3 

(0.38) 

FPIF 

2 

0 

0 

2 

1 

0 

3.33 

0.47 

0.22 

3 

(0.38) 

CPIF 

2 

0 

0 

3 

0 

0 

3.00 

0.00 

0.00 

3 

(0.61) 

CPAF 

2 

0 

0 

3 

0 

0 

3.00 

0.00 

0.00 

3 

(0.61) 

CPFF 

2 

0 

0 

3 

0 

0 

3.00 

0.00 

0.00 

3 

(0.61) 

T&M 

2 

0 

0 

2 

1 

0 

3.33 

0.47 

0.22 

3 

(0.38) 

Labor  Hour 

2 

0 

0 

2 

1 

0 

3.33 

0.47 

0.22 

3 

(0.38) 

Question  4 

Pre-Award 

1 

0 

1 

3 

0 

0 

2.75 

0.43 

0.19 

3 

(1.71) 

Post-Award 

1 

0 

1 

2 

1 

0 

3.00 

0.71 

0.50 

3 

(1.12) 
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Question  5 

No  Response 

#1 

#2 

#3 

#4 

#5 

Mean 

Std  Dev 

Variation 

Median 

Skew 

5a 

1 

0 

2 

1 

1 

0 

2.75 

0.83 

0.69 

3 

(0.55) 

5b 

1 

0 

3 

1 

0 

0 

2.25 

0.43 

0.19 

2 

(1.29) 

5c 

2 

1 

2 

0 

0 

0 

1.67 

0.47 

0.22 

2 

0.00 

5d 

1 

1 

3 

0 

0 

0 

1.75 

0.43 

0.19 

2 

(126) 

5e 

2 

0 

1 

1 

1 

0 

3.00 

0.82 

0.67 

3 

0.05 

5f 

3 

0 

2 

0 

0 

0 

2.00 

0.00 

0.00 

2 

0.61 

Question  6 

6a 

2 

0 

2 

1 

0 

0 

2.33 

0.47 

0.22 

2 

(0.17) 

6b 

3 

0 

1 

0 

1 

0 

3.00 

1.00 

1.00 

3 

1.26 

6c 

1 

1 

1 

2 

0 

0 

2.25 

0.83 

0.69 

3 

(0.54) 

Sd 

2 

0 

2 

1 

0 

0 

2.33 

0.47 

0.22 

2 

(0.17) 

Se 

2 

0 

2 

1 

0 

0 

2.33 

0.47 

0.22 

2 

(0.17) 

6f 

1 

2 

2 

0 

0 

0 

1.50 

0.50 

0.25 

2 

(0.51) 

6g 

3 

0 

1 

1 

0 

0 

2.50 

0.50 

0.25 

3 

0.88 

Question  7 

7a 

1 

1 

2 

1 

0 

0 

2.00 

0.71 

0.50 

2 

(0.40) 

7b 

1 

1 

3 

0 

0 

0 

1.75 

0.43 

0.19 

2 

(1.26) 

7c 

2 

1 

2 

0 

0 

0 

1.67 

0.47 

0.22 

2 

0.00 

Question  8 

8a 

2 

1 

2 

0 

1 

0 

2.25 

0.94 

0.89 

2 

0.51 

8b 

2 

0 

2 

0 

1 

0 

2.67 

0.94 

0.89 

2 

0.51 

8c 

2 

0 

3 

0 

0 

0 

2.00 

0.00 

0.00 

2 

(0-61) 

8d 

2 

0 

2 

1 

0 

0 

2.33 

0.47 

0.22 

2 

(0.17) 

Question  9 

- 

9 

2 

0 

1 

0 

1 

1 

3.67 

1.25 

1.56 

4 

0.23 

Question  10 

10a 

1 

0 

3 

0 

1 

0 

2.50 

0.87 

0.75 

2 

0.00 

10b 

1 

0 

1 

1 

0 

1 

3.33 

1.30 

1.69 

4 

(0.52) 

10c 

1 

0 

2 

1 

0 

1 

3.00 

1.22 

1.50 

3 

0.27 

10d 

2 

0 

0 

1 

1 

1 

4.00 

0.82 

0.67 

4 

(0.20) 
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Auditor  Group  Data 


No  Response 

#1 

#2 

#3 

#4 

#5 

Mean 

Std  Dev 

Variation 

Median 

Skew 

Question  1 

Benchmarking 

14 

1 

8 

16 

7 

0 

2.91 

0.76 

0.58 

3 

(0.33) 

Validating 

10 

2 

12 

15 

5 

2 

2.81 

0.94 

0.88 

3 

(0.22) 

Program  Est 

14 

1 

10 

18 

3 

0 

2.72 

0.67 

0.45 

3 

(0.37) 

Budgeting 

17 

1 

13 

11 

4 

0 

2.62 

0.76 

0.58 

3 

0.02 

ICE 

33 

1 

4 

4 

4 

0 

2.85 

0.95 

0.90 

3 

1.40 

Software 

22 

1 

6 

9 

8 

0 

3.00 

0.87 

0.75 

3 

0.30 

Repair  Parts 

18 

1 

7 

12 

8 

0 

2.96 

0.82 

0.68 

3 

(0.01) 

R&D 

15 

0 

4 

9 

17 

1 

3.48 

0.76 

0.57 

4 

(0.41) 

Service 

16 

1 

8 

8 

13 

0 

3.10 

0.91 

0.82 

3 

(0.13) 

Hardware 

14 

2 

15 

10 

5 

0 

2.56 

0.83 

0.68 

2 

(0.07) 

Construction 

17 

1 

10 

14 

4 

0 

2.72 

0.74 

0.54 

3 

(0.07) 

Commodity 

23 

0 

3 

12 

7 

1 

3.26 

0.74 

0.54 

3 

0.28 

Question  2 

CE 

14 

1 

8 

9 

14 

0 

3.13 

0.89 

0.80 

3 

(0.29) 

PD&RR 

12 

1 

10 

17 

6 

0 

2.82 

0.75 

0.56 

3 

(0.46) 

E&MD 

14 

1 

9 

18 

4 

0 

2.78 

0.70 

0.48 

3 

(0.37) 

PFD&OS 

15 

1 

13 

14 

3 

0 

2.61 

0.70 

0.50 

3 

(0.20) 

Question  3 

FFP 

15 

0 

14 

12 

5 

0 

2.71 

0.73 

0.53 

3 

(0.17) 

FPE 

18 

0 

12 

11 

5 

0 

2.75 

0.74 

0.54 

3 

0.03 

FPIS 

18 

0 

10 

13 

5 

0 

2.82 

0.71 

0.50 

3 

(0.03) 

FPR(P) 

17 

0 

13 

12 

4 

0 

2.69 

0.70 

0.49 

3 

(0.06) 

FPR(R) 

16 

0 

11 

14 

5 

0 

2.80 

0.70 

0.49 

3 

(0.17) 

FPIF 

15 

0 

13 

14 

4 

0 

2.71 

0.68 

0.46 

3 

(0.23) 

CPIF 

15 

0 

9 

16 

6 

0 

2.90 

0.69 

0.47 

3 

(0.30) 

CPAF 

16 

1 

8 

14 

7 

0 

2.90 

0.79 

0.62 

3 

(0.15) 

CPFF 

15 

1 

8 

16 

6 

0 

2.87 

0.75 

0.56 

3 

(0.26) 

T&M 

15 

0 

5 

11 

15 

0 

3.32 

0.74 

0.54 

3 

(0.39) 

Labor  Hour 

16 

0 

4 

13 

13 

0 

3.30 

0.69 

0.48 

3 

(0.34) 

Question  4 

Pre-Award 

11 

1 

17 

15 

2 

0 

2.51 

0.65 

0.42 

2 

(0.51) 

Post-Award 

11 

0 

1 

12 

17 

5 

3.74 

0.73 

0.53 

4 

(0.79) 
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Question  5 

No  Response 

#1 

#2 

#3 

#4 

#5 

Mean 

Std  Dev 

Variation 

Median 

Skew 

5a 

8 

6 

8 

19 

5 

0 

2.61 

0.90 

0.82 

3 

(0.49) 

5b 

7 

5 

21 

10 

3 

0 

2.28 

0.78 

0.61 

2 

(0.28) 

5c 

7 

4 

26 

8 

1 

0 

2.15 

0.62 

0.39 

2 

(0.54) 

5d 

8 

2 

21 

13 

2 

0 

2.39 

0.67 

0.45 

2 

(0.57) 

5e 

12 

0 

8 

19 

7 

0 

2.97 

0.66 

0.44 

3 

(0.59) 

5f 

7 

4 

22 

9 

4 

0 

2.33 

0.80 

0.63 

2 

(0.25) 

Question  6 

6a 

7 

3 

25 

9 

2 

0 

2.26 

0.67 

0.45 

2 

(0.46) 

6b 

10 

2 

11 

20 

3 

0 

2.67 

0.71 

0.50 

3 

(0.64) 

6c 

10 

3 

17 

9 

7 

0 

2.56 

0.90 

0.80 

2 

(0.18) 

6d 

12 

6 

20 

8 

0 

0 

2.06 

0.64 

0.41 

2 

(0.29) 

6e 

12 

6 

15 

11 

2 

0 

2.26 

0.82 

0.67 

2 

(0.08) 

6f 

9 

10 

20 

6 

1 

0 

1.95 

0.73 

0.54 

2 

(0.05) 

6g 

13 

1 

14 

14 

4 

0 

2.64 

0.73 

0.53 

3 

(0.30) 

Question  7 

7a 

6 

18 

19 

2 

1 

0 

1.65 

0.69 

0.48 

2 

0.32 

7b 

6 

13 

25 

1 

1 

0 

1.75 

0.62 

0.39 

2 

(0-07) 

7c 

6 

18 

21 

0 

1 

0 

1.60 

0.62 

0.39 

2 

0.24 

Question  8 

8a 

8 

11 

14 

9 

4 

0 

2.16 

0.96 

0.92 

2 

0.12 

8b 

8 

13 

11 

9 

5 

0 

2.16 

1.04 

1.08 

2 

0.22 

8c 

6 

15 

21 

3 

1 

0 

1.75 

0.70 

0.49 

2 

0.13 

8d 

6 

12 

19 

8 

1 

0 

1.95 

0.77 

0.60 

2 

(0.07) 

Question  9 

9 

16 

5 

17 

7 

1 

0 

2.13 

0.72 

0.52 

2 

0.10 

Question  1 0 

10a 

8 

2 

21 

12 

3 

0 

2.42 

0.71 

0.51 

2 

(0.48) 

10b 

13 

0 

9 

22 

2 

0 

2.79 

0.54 

0.29 

3 

(0.62) 

10c 

10 

1 

17 

15 

3 

0 

2.56 

0.68 

0.47 

3 

(0.53) 

10d 

10 

0 

15 

19 

2 

0 

2.64 

0.58 

0.34 

3 

(0.75) 
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Industry  Group  Data 


No  Response 

#1 

#2\ 

#3 

#4 

#5 

Mean 

Std  Dev 

Variation 

Median 

Skew 

Question  1 

Benchmarking 

14 

7 

6 

0 

2 

0 

1.80 

0.98 

0.96 

2 

1.31 

Validating 

10 

8 

5 

4 

1 

1 

2.05 

1.15 

1.31 

2 

0.93 

Program  Est 

12 

11 

4 

2 

0 

0 

1.47 

0.70 

0.48 

1 

0.89 

Budgeting 

2 

17 

6 

3 

1 

0 

1.56 

0.83 

0.69 

1 

1.08 

ICE 

5 

10 

7 

6 

1 

0 

1.92 

0.91 

0.83 

2 

0.26 

Software 

7 

13 

4 

3 

1 

1 

1.77 

1.13 

1.27 

1 

1.24 

Repair  Parts 

9 

3 

5 

8 

4 

0 

2.65 

0.96 

0.93 

3 

(0.03) 

R&D 

7 

7 

9 

4 

2 

0 

2.05 

0.93 

0.86 

2 

0.32 

Service 

8 

2 

4 

7 

8 

0 

3.00 

0.98 

0.95 

3 

(0.30) 

Hardware 

8 

9 

5 

4 

3 

0 

2.05 

1.09 

1.19 

2 

0.58 

Construction 

5 

7 

7 

6 

4 

0 

2.29 

1.06 

1.12 

2 

0.10 

Commodity 

13 

1 

1 

5 

9 

0 

3.38 

0.86 

0.73 

4 

0.07 

Question  2 

CE 

1 

13 

10 

4 

1 

0 

1.75 

0.83 

0.69 

2 

0.68 

PD&RR 

1 

13 

10 

3 

2 

0 

1.79 

0.90 

0.81 

2 

0.87 

E&MD 

2 

10 

9 

8 

0 

0 

1.93 

0.81 

0.66 

2 

(0.11) 

PFD&OS 

4 

8 

5 

9 

3 

0 

2.28 

1.03 

1.06 

2 

(0.02) 

Question  3 

FFP 

7 

9 

6 

5 

1 

2.05 

1.11 

1.23 

2 

0.72 

FPE 

11 

8 

5 

3 

1 

2.00 

1.15 

1.33 

2 

1.10 

FPIS 

10 

8 

5 

5 

0 

1.95 

0.94 

0.89 

2 

0.56 

FPR(P) 

10 

8 

6 

4 

0 

1.89 

0.91 

0.83 

2 

0.60 

FPR(R) 

10 

9 

5 

4 

0 

1.84 

0.93 

0.87 

2 

0.70 

FPIF 

9 

9 

5 

5 

0 

1.90 

0.94 

0.89 

2 

0.55 

CPIF 

9 

12 

3 

2 

3 

0 

1.80 

1.12 

1.26 

1 

1.08 

CPAF 

9 

12 

3 

2 

3 

0 

1.80 

1.12 

1.26 

1 

1.08 

CPFF 

9 

13 

3 

2 

2 

0 

1.65 

1.01 

1.03 

1 

1.20 

T&M 

8 

4 

4 

5 

4 

3 

2.90 

1.44 

2.09 

3 

0.29 

Labor  Hour 

11 

2 

4 

5 

4 

3 

3.11 

1.24 

1.54 

3 

0.30 

Question  4 

Pre-Award 

3 

13 

7 

6 

0 

0 

1.73 

0.81 

0.66 

2 

0.25 

Post-Award 

5 

9 

5 

5 

4 

1 

2.29 

1.24 

1.54 

2 

0.42 
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Question  5 

No  Response 

#1 

#2 

#3 

#4 

#5 

Mean 

Std  Dev 

Variation 

Median 

Skew 

5a 

0 

14 

7 

8 

0 

0 

1.79 

0.85 

0.72 

2 

0.43 

5b 

10 

7 

8 

3 

1 

0 

1.89 

0.85 

0.73 

2 

0.54 

5c 

0 

16 

9 

4 

0 

0 

1.59 

0.72 

0.52 

1 

0.85 

5d 

0 

14 

11 

2 

0 

2 

1.79 

1.06 

1.13 

2 

1.90 

5e 

11 

3 

7 

5 

3 

0 

2.44 

0.96 

0.91 

2 

0.32 

5f 

9 

7 

10 

2 

1 

0 

1.85 

0.79 

0.63 

2 

0.45 

Question  6 

6a 

3 

7 

11 

5 

3 

0 

2.15 

0.95 

0.90 

2 

0.14 

6b 

2 

4 

2 

16 

5 

0 

2.81 

0.90 

0.82 

3 

(1.00) 

6c 

12 

5 

8 

3 

1 

0 

2.00 

0.84 

0.71 

2 

0.58 

6d 

0 

5 

7 

13 

4 

0 

2.55 

0.93 

0.87 

3 

(0.29) 

Be 

1 

4 

11 

9 

4 

0 

2.46 

0.91 

0.82 

2 

(0.20) 

6f 

3 

5 

13 

6 

2 

0 

2.19 

0.83 

0.69 

2 

(0.13) 

6g 

12 

1 

7 

6 

3 

0 

2.65 

0.84 

0.70 

3 

0.21 

Question  7 

7a 

3 

9 

8 

6 

3 

0 

2.12 

1.01 

1.03 

2 

0.21 

7b 

2 

13 

10 

3 

1 

0 

1.70 

0.81 

0.65 

2 

0.65 

7c 

3 

13 

11 

1 

1 

0 

1.62 

0.74 

0.54 

2 

0.70 

Question  8 

8a 

4 

4 

12 

4 

5 

0 

2.40 

0.98 

0.96 

2 

(0.02) 

8b 

5 

4 

15 

2 

3 

0 

2.17 

0.85 

0.72 

2 

0.13 

8c 

4 

6 

14 

4 

1 

0 

2.00 

0.75 

0.56 

2 

(0.09) 

8d 

16 

1 

8 

3 

1 

0 

2.31 

0.72 

0.52 

2 

0.72 

Question  9 

9 

7 

2 

8 

8 

3 

1 

2.68 

0.97 

0.94 

3 

(0.06) 

Question  10 

10a 

1 

4 

12 

8 

2 

2 

2.50 

1.05 

1.11 

2 

0.45 

10b 

10 

0 

7 

11 

1 

0 

2.68 

0.57 

0.32 

3 

(0.32) 

10c 

2 

3 

9 

12 

3 

0 

2.56 

0.83 

0.69 

3 

(0.65) 

10d 

2 

3 

12 

8 

4 

0 

2.48 

0.88 

0.77 

2 

(0.31) 
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U.S.  Army  Group  Data 


No  Response 

#1 

#2 

#3 

#4 

#5^ 

Mean 

Std  Dev 

Variation 

Median 

Skew 

Question  1 

Benchmarking 

20 

1 

10 

17 

3 

0 

2.71 

0.68 

0.46 

3 

(0.06) 

Validating 

16 

2 

6 

21 

6 

0 

2.89 

0.75 

0.56 

3 

(0.34) 

Program  Est 

13 

4 

16 

17 

1 

0 

2.39 

0.71 

0.50 

2 

(0.39) 

Budgeting 

14 

3 

14 

18 

2 

0 

2.51 

0.72 

0.52 

3 

(0.35) 

ICE 

18 

3 

13 

15 

2 

0 

2.48 

0.74 

0.55 

3 

(0.06) 

Software 

18 

5 

8 

13 

7 

0 

2.67 

0.97 

0.95 

3 

0.09 

Repair  Parts 

14 

1 

6 

24 

6 

0 

2.95 

0.66 

0.43 

3 

(0.58) 

R&D 

13 

5 

8 

16 

8 

1 

2.79 

1.00 

1.01 

3 

(0.17) 

Service 

10 

3 

9 

15 

13 

1 

3.00 

0.96 

0.93 

3 

(0.49) 

Hardware 

9 

2 

10 

25 

5 

0 

2.79 

0.71 

0.50 

3 

(0.84) 

Construction 

17 

1 

13 

13 

7 

0 

2.76 

0.81 

0.65 

3 

(0.11) 

Commodity 

24 

0 

4 

18 

5 

0 

3.04 

0.58 

0.33 

3 

0.09 

Question  2 

CE 

11 

7 

11 

15 

5 

2 

2.60 

1.07 

1.14 

3 

0.01 

PD&RR 

11 

3 

17 

18 

2 

0 

2.48 

0.71 

0.50 

3 

(0.52) 

E&MD 

10 

2 

15 

21 

3 

0 

2.61 

0.69 

0.48 

3 

(0.67) 

PFD&OS 

10 

3 

11 

17 

10 

0 

2.83 

0.88 

0.78 

3 

(0.52) 

Question  3 

FFP 

13 

4 

9 

15 

9 

1 

2.84 

0.99 

0.98 

3 

(0.21) 

FPE 

18 

2 

7 

14 

9 

1 

3.00 

0.92 

0.85 

3 

(0.05) 

FPIS 

19 

1 

6 

20 

5 

0 

2.91 

0.68 

0.46 

3 

(0.19) 

FPR(P) 

18 

1 

6 

21 

5 

0 

2.91 

0.67 

0.45 

3 

(0.27) 

FPR(R) 

18 

1 

6 

19 

7 

0 

2.97 

0.72 

0.51 

3 

(0.24) 

FPIF 

19 

1 

9 

16 

6 

0 

2.84 

0.75 

0.57 

3 

(0.09) 

CPIF 

17 

2 

8 

16 

8 

0 

2.88 

0.83 

0.69 

3 

(0.17) 

CPAF 

17 

1 

12 

13 

8 

0 

2.82 

0.82 

0.67 

3 

(0.12) 

CPFF 

18 

3 

10 

14 

6 

0 

2.70 

0.87 

0.76 

3 

(0.00) 

T&M 

16 

1 

6 

17 

9 

2 

3.14 

0.87 

0.75 

3 

(0.23) 

Labor  Hour 

17 

1 

7 

12 

12 

2 

3.21 

0.93 

0.87 

3 

(0.15) 

Question  4 

Pre-Award 

8 

4 

10 

26 

3 

0 

2.65 

0.74 

0.55 

3 

(0.84) 

Post-Award 

10 

2 

5 

18 

15 

1 

3.20 

0.86 

0.74 

3 

(0.73) 
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Question  5 

No  Response 

#1 

#2 

#3 

#4 

#5 

Mean 

Std  Dev 

Variation 

Median 

Skew 

5a 

7 

3 

16 

15 

10 

0 

2.73 

0.89 

0.79 

3 

(0.52) 

5b 

8 

2 

19 

15 

7 

0 

2.63 

0.81 

0.65 

3 

(0.50) 

5c 

6 

6 

26 

7 

6 

0 

2.29 

0.86 

0.74 

2 

(0.04) 

5d 

7 

4 

27 

9 

4 

0 

2.30 

0.76 

0.57 

2 

(0.27) 

5e 

10 

2 

8 

17 

14 

0 

3.05 

0.85 

0.73 

3 

(0.68) 

5f 

8 

2 

24 

13 

4 

0 

2.44 

0.73 

0.53 

2 

(0.47) 

Question  6 

6a 

10 

8 

19 

8 

6 

0 

2.29 

0.94 

0.89 

2 

0.06 

6b 

8 

6 

15 

13 

9 

0 

2.58 

0.97 

0.94 

3 

(0.28) 

6c 

6 

3 

23 

12 

7 

0 

2.51 

0.83 

0.69 

2 

(0.36) 

6d 

6 

3 

25 

13 

4 

0 

2.40 

0.74 

0.55 

2 

(0.46) 

6e 

7 

7 

21 

14 

2 

0 

2.25 

0.77 

0.60 

2 

(0.39) 

6f 

9 

11 

24 

4 

3 

0 

1.98 

0.80 

0.64 

2 

0.18 

6g 

9 

5 

18 

16 

3 

0 

2.40 

0.79 

0.62 

2 

(0-42) 

Question  7 

7a 

6 

16 

25 

3 

1 

0 

1.76 

0.67 

0.45 

2 

0.04 

7b 

6 

16 

24 

4 

1 

0 

1.78 

0.70 

0.48 

2 

0.06 

7c 

6 

17 

27 

0 

1 

0 

1.67 

0.60 

0.36 

2 

(0.03) 

Question  8 

8a 

7 

6 

30 

6 

•  2 

0 

2.09 

0.67 

0.45 

2 

(0.30) 

8b 

6 

6 

30 

8 

1 

0 

2.09 

0.63 

0.39 

2 

(0.53) 

8c 

7 

8 

32 

2 

2 

0 

1.95 

0.64 

0.41 

2 

(0.16) 

8d 

11 

5 

30 

4 

1 

0 

2.03 

0.57 

0.32 

2 

(0.39) 

Question  9 

9 

18 

3 

12 

13 

5 

0 

2.61 

0.85 

0.72 

3 

0.03 

Question  10 

10a 

7 

1 

17 

23 

3 

0 

2.64 

0.64 

0.41 

3 

(0.96) 

10b 

11 

0 

10 

23 

7 

0 

2.93 

0.65 

0.42 

3 

(0.77) 

10c 

8 

2 

14 

20 

7 

0 

2.74 

0.78 

0.61 

3 

(0.70) 

10d 

10 

7 

13 

16 

5 

0 

2.46 

0.91 

0.83 

3 

(0.25) 
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DCMC  Group  Data 


No  Response 

#1 

#2 

#3 

#4 

#5 

Std  Dev 

Variation 

Median 

Skew 

Question  1 

Benchmarking 

12 

0 

8 

9 

5 

1 

2.96 

0.86 

0.74 

3 

(0.04) 

Validating 

11 

1 

9 

11 

3 

0 

2.67 

0.75 

0.56 

3 

(0.22) 

Program  Est 

10 

3 

13 

9 

0 

0 

2.24 

0.65 

0.42 

2 

(0.31) 

Budgeting 

9 

6 

12 

8 

0 

0 

2.08 

0.73 

0.53 

2 

(0.18) 

ICE 

18 

5 

4 

5 

3 

0 

2.35 

1.08 

1.17 

2 

0.85 

Software 

13 

3 

7 

8 

4 

0 

2.59 

0.94 

0.88 

3 

0.17 

Repair  Parts 

10 

1 

7 

13 

4 

0 

2.80 

0.75 

0.56 

3 

(0.38) 

R&D 

10 

1 

10 

6 

7 

1 

2.88 

0.99 

0.99 

3 

(0.05) 

Service 

12 

1 

4 

10 

8 

0 

3.09 

0.83 

0.69 

3 

(0.22) 

Hardware 

11 

2 

12 

9 

1 

0 

2.38 

0.70 

0.48 

2 

(0.17) 

Construction 

18 

1 

7 

4 

5 

0 

2.76 

0.94 

0.89 

3 

0.59 

Commodity 

20 

0 

3 

5 

7 

0 

3.27 

0.77 

0.60 

3 

0.55 

Question  2 

CE 

12 

5 

6 

5 

7 

0 

2.61 

1.11 

1.23 

3 

0.25 

PD&RR 

10 

4 

9 

6 

5 

1 

2.60 

1.10 

1.20 

2 

0.20 

E&MD 

10 

3 

13 

7 

2 

0 

2.32 

0.75 

0.56 

2 

(0.13) 

PFD&OS 

9 

3 

1 

1 

0 

0 

1.60 

0.84 

0.71 

3 

(0.20) 

Question  3 

FFP 

8 

12 

15 

8 

2 

0 

2.00 

0.86 

0.74 

2 

0.01 

FPE 

16 

10 

7 

10 

2 

0 

2.14 

0.88 

0.78 

3 

0.41 

FPIS 

17 

9 

8 

11 

0 

0 

2.07 

0.68 

0.46 

3 

0.30 

FPR(P) 

14 

6 

11 

9 

0 

0 

2.12 

0.70 

0.49 

2 

0.08 

FPR(R) 

15 

6 

8 

10 

1 

0 

2.24 

0.80 

0.64 

3 

0.25 

FPIF 

13 

6 

9 

11 

1 

0 

2.26 

0.78 

0.61 

3 

0.05 

CPIF 

10 

7 

8 

12 

3 

0 

2.37 

0.90 

0.81 

3 

(0.11) 

CPAF 

13 

7 

4 

14 

2 

0 

2.41 

0.89 

0.79 

3 

0.03 

CPFF 

11 

7 

6 

14 

2 

0 

2.38 

0.87 

0.75 

3 

(0.12) 

T&M 

12 

3 

9 

9 

5 

2 

2.79 

1.08 

1.17 

3 

0.13 

Labor  Hour 

12 

3 

11 

7 

5 

2 

2.71 

1.12 

1.24 

3 

0.23 

Question  4 

Pre-Award 

8 

5 

12 

7 

3 

0 

2.30 

0.90 

0.80 

2 

(0.01) 

Post-Award 

9 

4 

6 

11 

4 

1 

2.69 

1.03 

1.06 

3 

(0.06) 
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Question  5 

No  Response 

#1 

#2 

#3 

#4 

#5 

Std  Dev 

Variation 

Median 

Skew 

5a 

9 

3 

11 

9 

3 

0 

2.46 

0.84 

0.71 

2 

(0.16) 

5b 

10 

2 

14 

7 

2 

0 

2.36 

0.74 

0.55 

2 

(0.12) 

5c 

7 

8 

14 

5 

1 

0 

1.96 

0.78 

0.61 

2 

0.03 

5d 

7 

3 

17 

7 

1 

0 

2.21 

0.67 

0.45 

2 

(0.39) 

5e 

14 

0 

7 

10 

4 

0 

2.86 

0.71 

0.50 

3 

(0.01) 

5f 

9 

2 

16 

6 

2 

0 

2.31 

0.72 

0.52 

2 

(0.16) 

Question  6 

6a 

11 

2 

17 

5 

0 

0 

2.13 

0.53 

0.28 

2 

(0.31) 

6b 

11 

1 

9 

10 

4 

0 

2.71 

0.79 

0.62 

3 

(0.18) 

6c 

10 

2 

15 

8 

0 

0 

2.24 

0.59 

0.34 

2 

(0.39) 

6d 

9 

3 

13 

8 

2 

0 

2.35 

0.78 

0.61 

2 

(0.16) 

6e 

9 

3 

12 

8 

3 

0 

2.42 

0.84 

0.71 

2 

(0.12) 

6f 

10 

4 

14 

6 

1 

0 

2.16 

0.73 

0.53 

2 

(0.05) 

6g 

12 

3 

10 

8 

2 

0 

2.39 

0.82 

0.67 

2 

0.07 

Question  7 

7a 

6 

12 

15 

1 

1 

0 

1.69 

0.70 

0.49 

2 

0.31 

7b 

6 

8 

17 

4 

0 

0 

1.86 

0.63 

0.39 

2 

(0.37) 

7c 

7 

9 

16 

3 

0 

0 

1.79 

0.62 

0.38 

2 

(0.26) 

Question  8 

8a 

7 

2 

18 

5 

3 

0 

2.32 

0.76 

0.58 

2 

(0.18) 

8b 

8 

4 

17 

3 

3 

0 

2.19 

0.82 

0.67 

2 

0.09 

8c 

7 

9 

15 

3 

1 

0 

1.86 

0.74 

0.55 

2 

0.13 

8d 

9 

5 

15 

4 

2 

0 

2.12 

0.80 

0.64 

2 

0.11 

Question  9 

9 

10 

2 

12 

7 

4 

0 

2.52 

0.85 

0.73 

2 

(0.06) 

Question  10 

10a 

8 

2 

12 

10 

3 

0 

2.52 

0.79 

0.62 

2 

(0.33) 

10b 

13 

0 

7 

12 

3 

0 

2.82 

0.65 

0.42 

3 

(0.16) 

10c 

6 

1 

15 

12 

1 

0 

2.45 

0.62 

0.39 

2 

(0.77) 

10d 

7 

4 

11 

11 

2 

0 

2.39 

0.82 

0.67 

2 

(0.34) 
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DCAA  Group  Data 


No  Response 

#1 

#2 

#3 

#4 

#5 

Mean 

Std  Dev 

Variation 

Median 

Skew 

Question  1 

Benchmarking 

14 

1 

8 

17 

7 

0 

2.91 

0.75 

0.57 

3 

(0.37) 

Validating 

9 

2 

12 

17 

5 

2 

2.82 

0.91 

0.83 

3 

(0.32) 

Program  Est 

14 

1 

10 

18 

4 

0 

2.76 

0.70 

0.49 

3 

(0.37) 

Budgeting 

17 

1 

13 

12 

4 

0 

2.63 

0.75 

0.57 

3 

(0.03) 

ICE 

34 

1 

4 

4 

4 

0 

2.85 

0.95 

0.90 

3 

1.43 

Software 

22 

1 

6 

9 

8 

1 

3.08 

0.93 

0.87 

3 

0.32 

Repair  Parts 

18 

1 

7 

13 

8 

0 

2.97 

0.81 

0.65 

3 

(0.05) 

R&D 

15 

0 

4 

9 

17 

2 

3.53 

0.79 

0.62 

4 

(0.41) 

Service 

16 

1 

8 

8 

14 

0 

3.13 

0.91 

0.82 

3 

(0.16) 

Hardware 

14 

2 

15 

11 

5 

0 

2.58 

0.82 

0.67 

2 

(0.11) 

Construction 

17 

1 

10 

14 

5 

0 

2.77 

0.76 

0.58 

3 

(0.08) 

Commodity 

24 

0 

3 

12 

7 

1 

3.26 

0.74 

0.54 

3 

0.32 

Question  2 

CE 

14 

1 

9 

9 

14 

0 

3.09 

0.90 

0.81 

3 

(0.29) 

PD&RR 

11 

1 

10 

19 

6 

0 

2.83 

0.73 

0.53 

3 

(0.58) 

E&MD 

14 

1 

9 

19 

4 

0 

2.79 

0.69 

0.47 

3 

(0.40) 

PFD&OS 

15 

1 

13 

14 

4 

0 

2.66 

0.73 

0.54 

3 

(0.20) 

Question  3 

FFP 

15 

0 

14 

12 

6 

0 

2.75 

0.75 

0.56 

3 

(0.18) 

FPE 

18 

0 

12 

11 

6 

0 

2.79 

0.76 

0.58 

3 

0.01 

FPIS 

18 

0 

10 

13 

6 

0 

2.86 

0.73 

0.53 

3 

(0.05) 

FPR(P) 

17 

0 

13 

12 

5 

0 

2.73 

0.73 

0.53 

3 

(0.07) 

FPR(R) 

16 

0 

11 

14 

6 

0 

2.84 

0.72 

0.52 

3 

(0.19) 

FPIF 

15 

0 

13 

14 

5 

0 

2.75 

0.71 

0.50 

3 

(0.24) 

CPIF 

14 

0 

9 

18 

6 

0 

2.91 

0.67 

0.45 

3 

(0.43) 

CPAF 

16 

1 

8 

15 

7 

0 

2.90 

0.78 

0.60 

3 

(0.19) 

CPFF 

15 

1 

8 

17 

6 

0 

2.88 

0.74 

0.55 

3 

(0.30) 

T&M 

15 

0 

5 

11 

16 

0 

3.34 

0.73 

0.54 

4 

(0.43) 

Labor  Hour 

16 

0 

4 

13 

14 

0 

3.32 

0.69 

0.48 

3 

(0.37) 

Question  4 

Pre-Award 

10 

1 

17 

17 

2 

0 

2.54 

0.64 

0.41 

3 

(0.63) 

Post-Award 

11 

0 

1 

12 

18 

5 

3.75 

0.72 

0.52 

4 

(0.83) 
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Question  5 

No  Response 

#1 

#2 

#3 

#4 

#5 

Mean 

Std  Dev 

Variation 

Median 

Skew 

5a 

7 

6 

9 

19 

6 

0 

2.63 

0.91 

0.83 

3 

(0.53) 

5b 

6 

5 

22 

11 

3 

0 

2.29 

0.77 

0.60 

2 

(0.34) 

5c 

7 

4 

27 

8 

1 

0 

2.15 

0.61 

0.38 

2 

(0.56) 

5d 

7 

3 

22 

13 

2 

0 

2.35 

0.69 

0.48 

2 

(0.57) 

5e 

11 

0 

9 

20 

7 

0 

2.94 

0.66 

0.44 

3 

(0.68) 

5f 

7 

4 

23 

9 

4 

0 

2.33 

0.79 

0.62 

2 

(0.25) 

Question  6 

6a 

6 

4 

25 

10 

2 

0 

2.24 

0.69 

0.48 

2 

(0.46) 

6b 

9 

2 

13 

20 

3 

0 

2.63 

0.70 

0.50 

3 

(0.70) 

6c 

9 

3 

17 

10 

8 

0 

2.61 

0.90 

0.82 

2 

(0.26) 

6d 

11 

6 

22 

8 

0 

0 

2.06 

0.62 

0.39 

2 

(0.39) 

Be 

11 

6 

17 

11 

2 

0 

2.25 

0.79 

0.63 

2 

(0.16) 

6f 

8 

12 

20 

6 

1 

0 

1.90 

0.74 

0.55 

2 

(0.01) 

6g 

13 

1 

14 

15 

4 

0 

2.65 

0.72 

0.52 

3 

(0.34) 

Question  7 

7a 

5 

20 

19 

2 

1 

0 

1.62 

0.69 

0.47 

2 

0.42 

7b 

5 

15 

25 

1 

1 

0 

1.71 

0.63 

0.39 

2 

0.02 

7c 

5 

20 

21 

0 

1 

0 

1.57 

0.62 

0.39 

2 

0.34 

Question  8 

8a 

7 

12 

14 

9 

5 

0 

2.18 

1.00 

0.99 

2 

0.15 

8b 

7 

14 

11 

9 

6 

0 

2.18 

1.07 

1.14 

2 

0.22 

8c 

5 

16 

22 

3 

1 

0 

1.74 

0.69 

0.48 

2 

0.15 

3d 

5 

12 

21 

8 

1 

0 

1.95 

0.75 

0.57 

2 

(0.10) 

Question  9 

9 

15 

5 

17 

8 

1 

1 

2.25 

0.87 

0.75 

2 

0.31 

Question  10 

10a 

7 

2 

23 

12 

3 

0 

2.40 

0.70 

0.49 

2 

(0.52) 

10b 

12 

0 

10 

22 

2 

1 

2.83 

0.65 

0.43 

3 

(0.52) 

10c 

9 

2 

17 

15 

3 

1 

2.58 

0.82 

0.66 

3 

(0.30) 

10d 

9 

0 

15 

20 

2 

1 

2.71 

0.68 

0.47 

3 

(0.59) 
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NASA  Group  Data 


No  Response 

#1 

#2 

#3 

#4 

#5 

Mean 

Std  Dev 

Variation 

Median 

Skew 

Question  1 

Benchmarking 

3 

3 

1 

2 

0 

0 

1.83 

0.90 

0.81 

2 

0.57 

Validating 

1 

2 

3 

3 

0 

0 

2.13 

0.78 

0.61 

2 

(0.55) 

Program  Est 

2 

5 

1 

0 

1 

0 

1.57 

1.05 

1.10 

1 

1.68 

Budgeting 

0 

4 

3 

2 

0 

0 

1.78 

0.79 

0.62 

2 

0.50 

ICE 

3 

3 

2 

1 

0 

0 

1.67 

0.75 

0.56 

2 

0.55 

Software 

3 

2 

2 

1 

1 

0 

2.17 

1.07 

1.14 

2 

0.64 

Repair  Parts 

3 

1 

2 

2 

1 

0 

2.50 

0.96 

0.92 

3 

0.16 

R&D 

2 

3 

1 

1 

1 

0 

2.00 

1.12 

1.27 

2 

0.42 

Service 

1 

1 

2 

3 

2 

0 

2.75 

0.97 

0.94 

3 

(0.66) 

Hardware 

1 

3 

2 

2 

0 

0 

1.86 

1.05 

1.11 

2 

0.26 

Construction 

2 

1 

2 

4 

0 

0 

2.43 

0.88 

0.78 

3 

(0.56) 

Commodity 

3 

1 

1 

1 

3 

0 

3.00 

1.15 

1.33 

4 

0.00 

Question  2 

CE 

1 

6 

1 

0 

1 

0 

1.50 

1.00 

1.00 

1 

1.92 

PD&RR 

2 

5 

1 

0 

1 

0 

1.57 

1.05 

1.10 

1 

1.68 

E&MD 

0 

1 

4 

4 

0 

0 

2.33 

0.67 

0.44 

2 

(0.61) 

PFD&OS 

1 

0 

2 

5 

1 

0 

2.88 

0.60 

0.36 

3 

(1.51) 

Question  3 

FFP 

5 

2 

1 

0 

0 

2.00 

0.71 

0.50 

2 

0.87 

FPE 

6 

2 

0 

0 

0 

1.67 

0.47 

0.22 

2 

1.19 

FPIS 

5 

2 

1 

0 

0 

2.00 

0.71 

0.50 

2 

0.87 

FPR(P) 

6 

2 

0 

0 

0 

1.67 

0.47 

0.22 

2 

1.19 

FPR(R) 

6 

2 

0 

0 

0 

1.67 

0.47 

0.22 

2 

1.19 

FPIF 

5 

2 

1 

0 

0 

2.00 

0.71 

0.50 

2 

0.87 

CPIF 

5 

2 

0 

0 

2.25 

1.09 

1.19 

2 

1.36 

CPAF 

5 

2 

0 

0 

2.25 

1.09 

1.19 

2 

1.36 

CPFF 

5 

2 

0 

0 

2.25 

1.09 

1.19 

2 

1.36 

T&M 

6 

0 

1 

1 

0 

3.00 

0.82 

0.67 

3 

1.22 

Labor  Hour 

6 

0 

1 

1 

0 

3.00 

0.82 

0.67 

3 

1.22 

Question  4 

Pre-Award 

1 

4 

1 

2 

0 

0 

1.71 

0.93 

0.86 

2 

0.15 

Post-Award 

2 

3 

1 

1 

2 

0 

2.29 

1.28 

1.63 

2 

0.47 
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Question  5 

No  Response 

#1 

#2 

#3 

#4 

#5 

Mean 

Std  Dev 

Variation 

Median 

Skew 

5a 

1 

1 

3 

3 

1 

0 

2.50 

0.87 

0.75 

3 

(0.54) 

5b 

2 

1 

4 

2 

0 

0 

2.14 

0.64 

0.41 

2 

(0.54) 

5c 

1 

5 

2 

1 

0 

0 

1.50 

0.71 

0.50 

1 

0.66 

5d 

1 

5 

2 

1 

0 

0 

1.50 

0.71 

0.50 

1 

0.66 

5e 

2 

1 

3 

2 

1 

0 

2.43 

0.90 

0.82 

2 

(0.13) 

5f 

2 

0 

6 

1 

0 

0 

2.14 

0.35 

0.12 

2 

(1.07) 

Question  6 

6a 

1 

3 

4 

1 

0 

0 

1.75 

0.66 

0.44 

2 

(0.21) 

6b 

0 

1 

1 

6 

0 

0 

2.63 

0.70 

0.48 

3 

(1.53) 

6c 

4 

1 

4 

0 

0 

0 

1.80 

0.40 

0.16 

2 

0.00 

6d 

1 

2 

3 

2 

1 

0 

2.25 

0.97 

0.94 

2 

0.00 

Be 

1 

0 

3 

3 

1 

0 

2.71 

0.78 

0.61 

3 

(0.93) 

6f 

1 

3 

3 

2 

0 

0 

1.88 

0.78 

0.61 

2 

(0.11) 

6g 

2 

0 

2 

4 

1 

0 

2.86 

0.64 

0.41 

3 

(0.86) 

Question  7 

7a 

1 

4 

2 

1 

1 

0 

1.88 

1.05 

1.11 

2 

0.82 

7b 

1 

3 

4 

0 

0 

0 

1.57 

0.50 

0.25 

2 

(0.61) 

7c 

1 

3 

4 

1 

0 

0 

1.75 

0.66 

0.44 

2 

(0.21) 

Question  8 

8a 

2 

0 

4 

3 

0 

0 

2.43 

0.49 

0.24 

2 

(0.95) 

8b 

2 

0 

3 

4 

0 

0 

2.57 

0.49 

0.24 

3 

(1.05) 

8c 

2 

1 

4 

2 

0 

0 

2.14 

0.64 

0.41 

2 

(0.54) 

8d 

3 

1 

4 

1 

0 

0 

2.00 

0.58 

0.33 

2 

(0.15) 

Question  9 

9 

2 

1 

6 

0 

0 

0 

1.86 

0.35 

0.12 

2 

(1.19) 

Question  10 

10a 

1 

0 

3 

2 

3 

0 

3.00 

0.87 

0.75 

3 

(0.88) 

10b 

2 

0 

2 

3 

2 

0 

3.00 

0.76 

0.57 

3 

(0.73) 

10c 

1 

1 

2 

3 

2 

0 

2.75 

0.97 

0.94 

3 

(0.66) 

10d 

1 

1 

3 

3 

1 

0 

2.50 

0.87 

0.75 

3 

(0.54) 
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U.S.  Navy  Group  Data 


No  Response 

#1 

#2 

#3 

#4 

#5 

Mean 

Std  Dev 

Variation 

Median 

Skew 

Question  1 

Benchmarking 

10 

0 

6 

5 

1 

0 

2.58 

0.64 

0.41 

3 

0.20 

Validating 

3 

1 

9 

8 

1 

0 

2.47 

0.68 

0.46 

2 

(0.79) 

Program  Est 

3 

3 

9 

7 

0 

0 

2.21 

0.69 

0.48 

2 

(0.69) 

Budgeting 

3 

3 

12 

3 

1 

0 

2.11 

0.72 

0.52 

2 

(0.22) 

ICE 

8 

2 

5 

4 

3 

0 

2.57 

0.98 

0.96 

3 

0.23 

Software 

7 

1 

5 

6 

3 

0 

2.73 

0.85 

0.73 

3 

(0.12) 

Repair  Parts 

7 

0 

5 

5 

5 

0 

3.00 

0.82 

0.67 

3 

(0.24) 

R&D 

5 

1 

3 

11 

2 

0 

2.82 

0.71 

0.50 

3 

(0.73) 

Service 

6 

0 

2 

8 

6 

0 

3.25 

0.66 

0.44 

3 

(0.66) 

Hardware 

2 

1 

9 

8 

2 

0 

2.55 

0.74 

0.55 

3 

(0.72) 

Construction 

12 

0 

1 

8 

1 

0 

3.00 

0.45 

0.20 

3 

0.32 

Commodity 

12 

0 

1 

6 

3 

0 

3.20 

0.60 

0.36 

3 

0.38 

Question  2 

CE 

1 

5 

3 

9 

4 

0 

2.57 

1.05 

1.10 

3 

(0.45) 

PD&RR 

1 

2 

8 

9 

2 

0 

2.52 

0.79 

0.63 

3 

(0.61) 

E&MD 

0 

3 

6 

11 

2 

0 

2.55 

0.84 

0.70 

3 

(0.40) 

PFD&OS 

1 

1 

7 

7 

6 

0 

2.86 

0.89 

0.79 

3 

(0.65) 

Question  3 

FFP 

4 

0 

7 

8 

3 

0 

2.78 

0.71 

0.51 

3 

(0.71) 

FPE 

8 

0 

6 

5 

3 

0 

2.79 

0.77 

0.60 

3 

(0.03) 

FPIS 

9 

1 

5 

6 

1 

0 

2.54 

0.75 

0.56 

3 

0.11 

FPR(P) 

8 

1 

5 

7 

1 

0 

2.57 

0.73 

0.53 

3 

(0.08) 

FPR(R) 

9 

1 

6 

5 

1 

0 

2.46 

0.75 

0.56 

2 

0.17 

FPIF 

5 

0 

7 

8 

2 

0 

2.71 

0.67 

0.44 

3 

(0.61) 

CPIF 

6 

2 

6 

6 

2 

0 

2.50 

0.87 

0.75 

3 

(0.13) 

CPAF 

5 

2 

6 

8 

1 

0 

2.47 

0.78 

0.60 

3 

(0.43) 

CPFF 

4 

2 

6 

8 

2 

0 

2.56 

0.83 

0.69 

3 

(0.49) 

T&M 

9 

0 

1 

9 

3 

0 

3.15 

0.53 

0.28 

3 

(0.19) 

Labor  Hour 

8 

0 

3 

7 

4 

0 

3.07 

0.70 

0.49 

3 

(0.22) 

Question  4 

Pre-Award 

2 

3 

9 

5 

3 

0 

2.40 

0.92 

0.84 

2 

(0.18) 

Post-Award 

3 

1 

5 

6 

7 

0 

3.00 

0.92 

0.84 

3 

(0.76) 
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Question  5 

No  Response 

#1 

#2 

#3 

#4 

#5 

Mean 

Std  Dev 

Variation 

Median 

Skew 

5a 

1 

2 

9 

9 

1 

0 

2.43 

0.73 

0.53 

2 

(0.71) 

5b 

1 

2 

8 

10 

1 

0 

2.48 

0.73 

0.54 

3 

(0.83) 

5c 

0 

2 

13 

7 

0 

0 

2.23 

0.60 

0.36 

2 

(0.14) 

5d 

1 

2 

13 

3 

3 

0 

2.33 

0.84 

0.70 

2 

0.18 

5e 

3 

1 

4 

8 

6 

0 

3.00 

0.86 

0.74 

3 

(0.88) 

5f 

4 

1 

10 

6 

1 

0 

2.39 

0.68 

0.46 

2 

(0.55) 

Question  6 

6a 

2 

1 

12 

5 

2 

0 

2.40 

0.73 

0.54 

2 

(0.40) 

6b 

1 

0 

7 

12 

2 

0 

2.76 

0.61 

0.37 

3 

(1.25) 

6c 

2 

2 

13 

3 

2 

0 

2.25 

0.77 

0.59 

2 

(0.10) 

6d 

1 

5 

7 

6 

3 

0 

2.33 

0.99 

0.98 

2 

(0.03) 

Be 

1 

5 

8 

5 

3 

0 

2.29 

0.98 

0.97 

2 

0.08 

6f 

4 

1 

14 

1 

2 

0 

2.22 

0.71 

0.51 

2 

(0.08) 

6g 

3 

1 

8 

5 

5 

0 

2.74 

0.91 

0.83 

3 

(0.47) 

Question  7 

7a 

2 

4 

12 

4 

0 

0 

2.00 

0.63 

0.40 

2 

(0.64) 

7b 

1 

5 

16 

0 

0 

0 

1.76 

0.43 

0.18 

2 

(1.67) 

7c 

1 

7 

14 

0 

0 

0 

1.67 

0.47 

0.22 

2 

(1.15) 

Question  8 

8a 

6 

3 

7 

5 

1 

0 

2.25 

0.83 

0.69 

2 

(0.02) 

8b 

6 

2 

10 

1 

3 

0 

2.31 

0.92 

0.84 

2 

0.24 

8c 

5 

4 

10 

1 

2 

0 

2.06 

0.87 

0.76 

2 

0.33 

8d 

6 

2 

11 

2 

1 

0 

2.13 

0.70 

0.48 

2 

(0.02) 

Question  9 

9 

8 

2 

10 

2 

0 

0 

2.00 

0.53 

0.29 

2 

(0.10) 

Question  1 0 

10a 

3 

1 

7 

6 

5 

0 

2.79 

0.89 

0.80 

3 

(0.57) 

10b 

4 

1 

6 

9 

2 

0 

2.67 

0.75 

0.56 

3 

(0.69) 

10c 

5 

0 

6 

8 

3 

0 

2.82 

0.71 

0.50 

3 

(0.61) 

10d 

5 

3 

6 

6 

2 

0 

2.41 

0.91 

0.83 

2 

(0.14) 
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USAF  Group  Data 


No  Response 

#1 

#2 

#3 

#4 

#5 

Mean 

Std  Dev 

Variation 

Median 

Skew 

Question  1 

Benchmarking 

5 

2 

5 

5 

0 

0 

2.25 

0.72 

0.52 

2 

(0.23) 

Validating 

1 

1 

7 

7 

1 

0 

2.50 

0.71 

0.50 

3 

(0.83) 

Program  Est 

1 

5 

8 

3 

0 

0 

1.88 

0.70 

0.48 

2 

(0.24) 

Budgeting 

1 

5 

6 

4 

1 

0 

2.06 

0.90 

0.81 

2 

0.13 

ICE 

3 

3 

7 

4 

0 

0 

2.07 

0.70 

0.49 

2 

(0.44) 

Software 

3 

2 

9 

3 

0 

0 

2.07 

0.59 

0.35 

2 

(0.66) 

Repair  Parts 

5 

1 

1 

8 

2 

0 

2.92 

0.76 

0.58 

3 

(0.47) 

R&D 

3 

2 

9 

1 

2 

0 

2.21 

0.86 

0.74 

2 

0.13 

Service 

3 

1 

1 

7 

5 

0 

3.14 

0.83 

0.69 

3 

(0.95) 

Hardware 

2 

1 

5 

8 

1 

0 

2.60 

0.71 

0.51 

3 

(0.98) 

Construction 

7 

1 

4 

3 

2 

0 

2.60 

0.92 

0.84 

3 

0.31 

Commodity 

6 

1 

0 

3 

7 

0 

3.45 

0.89 

0.79 

4 

(0.32) 

Question  2 

CE 

2 

4 

9 

1 

1 

0 

1.93 

0.77 

0.60 

2 

0.23 

PD&RR 

2 

5 

6 

2 

2 

0 

2.07 

1.00 

1.00 

2 

0.38 

E&MD 

2 

4 

8 

2 

1 

0 

2.00 

0.82 

0.67 

2 

0.15 

PFD&OS 

1 

4 

5 

6 

1 

0 

2.25 

0.85 

0.72 

2 

(0.48) 

Question  3 

FFP 

0 

1 

4 

11 

1 

0 

2.71 

0.67 

0.44 

3 

(0.86) 

FPE 

3 

0 

3 

9 

2 

0 

2.93 

0.59 

0.35 

3 

(1.10) 

FPIS 

3 

0 

7 

5 

2 

0 

2.64 

0.72 

0.52 

3 

(0.60) 

FPR(P) 

3 

0 

7 

5 

2 

0 

2.64 

0.72 

0.52 

3 

(0.60) 

FPR(R) 

3 

0 

7 

5 

2 

0 

2.64 

0.72 

0.52 

3 

(0.60) 

FPIF 

3 

0 

9 

3 

2 

0 

2.50 

0.73 

0.54 

2 

(0.37) 

CPIF 

4 

2 

6 

3 

2 

0 

2.38 

0.92 

0.85 

2 

0.00 

CPAF 

2 

4 

7 

2 

2 

0 

2.13 

0.96 

0.92 

2 

0.25 

CPFF 

2 

4 

7 

1 

3 

0 

2.20 

1.05 

1.09 

2 

0.34 

T&M 

2 

1 

3 

2 

9 

0 

3.27 

1.00 

1.00 

4 

(1.02) 

Labor  Hour 

2 

1 

3 

3 

8 

0 

3.20 

0.98 

0.96 

4 

(0.97) 

Question  4 

Pre-Award 

0 

4 

5 

7 

1 

0 

2.29 

0.89 

0.80 

2 

(0.12) 

Post-Award 

1 

2 

6 

4 

4 

0 

2.63 

0.96 

0.91 

3 

(0.37) 
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Question  5 

No  Response 

#1 

#2 

#3 

#4 

#5 

Mean 

Std  Dev 

Variation 

Median 

Skew 

5a 

0 

4 

4 

5 

4 

0 

2.53 

1.09 

1.19 

3 

(0.08) 

5b 

0 

2 

7 

7 

1 

0 

2.41 

0.77 

0.60 

2 

(0.10) 

5c 

0 

6 

6 

5 

0 

0 

1.94 

0.80 

0.64 

2 

0.12 

5d 

0 

6 

10 

1 

0 

0 

1.71 

0.57 

0.33 

2 

0.11 

5e 

1 

1 

3 

9 

3 

0 

2.88 

0.78 

0.61 

3 

(1.18) 

5f 

1 

3 

10 

2 

1 

0 

2.06 

0.75 

0.56 

2 

0.13 

Question  6 

6a 

1 

4 

10 

2 

0 

0 

1.88 

0.60 

0.36 

2 

(0.56) 

6b 

1 

0 

2 

9 

5 

0 

3.19 

0.63 

0.40 

3 

(1.70) 

6c 

0 

5 

9 

3 

0 

0 

1.88 

0.68 

0.46 

2 

0.16 

6d 

2 

1 

7 

6 

1 

0 

2.47 

0.72 

0.52 

2 

(0.74) 

Be 

2 

1 

7 

5 

2 

0 

2.53 

0.81 

0.65 

2 

(0.52) 

6f 

0 

1 

12 

3 

1 

0 

2.24 

0.64 

0.42 

2 

1.16 

6g 

2 

1 

5 

6 

3 

0 

2.73 

0.85 

0.73 

3 

(0.69) 

Question  7 

7a 

0 

3 

13 

1 

0 

0 

1.88 

0.47 

0.22 

2 

(0.40) 

7b 

0 

5 

11 

1 

0 

0 

1.76 

0.55 

0.30 

2 

(0.08) 

7c 

0 

5 

12 

0 

0 

0 

1.71 

0.46 

0.21 

2 

(0.99) 

Question  8 

8a 

2 

4 

6 

2 

3 

0 

2.27 

1.06 

1.13 

2 

0.21 

8b 

2 

3 

8 

4 

0 

0 

2.07 

0.68 

0.46 

2 

(0.60) 

8c 

1 

7 

6 

2 

1 

0 

1.81 

0.88 

0.78 

2 

0.68 

8d 

1 

4 

8 

1 

3 

0 

2.19 

1.01 

1.03 

2 

0.44 

Question  9 

9 

1 

0 

11 

4 

1 

0 

2.38 

0.58 

0.33 

2 

(0.47) 

Question  1 0 

10a 

0 

3 

3 

8 

3 

0 

2.65 

0.97 

0.93 

3 

(0.46) 

10b 

1 

0 

6 

6 

3 

1 

2.94 

0.90 

0.81 

3 

(0.32) 

10c 

1 

0 

3 

9 

4 

0 

3.06 

0.66 

0.43 

3 

(1.48) 

10d 

1 

4 

6 

5 

1 

0 

2.19 

0.88 

0.78 

2 

(0.13) 
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Cost  Estimating  Community  Group  Data 


No  Response 

#1 

#2 

#3 

#4 

#5 

Mean 

Std  Dev 

Variation 

Median 

Skew 

Question  1 

Benchmarking 

10 

6 

10 

8 

6 

0 

2.47 

1.02 

1.05 

2 

0.05 

Validating 

2 

5 

12 

17 

3 

1 

2.55 

0.91 

0.83 

3 

(0.27) 

Program  Est 

5 

8 

10 

16 

1 

0 

2.29 

0.85 

0.72 

2 

(0.47) 

Budgeting 

4 

8 

13 

13 

2 

0 

2.25 

0.86 

0.74 

2 

(0.31) 

ICE 

15 

4 

9 

8 

4 

0 

2.48 

0.94 

0.89 

2 

0.25 

Software 

10 

6 

6 

11 

6 

1 

2.67 

1.11 

1.22 

3 

0.00 

Repair  Parts 

11 

2 

4 

16 

7 

0 

2.97 

0.81 

0.65 

3 

(0.46) 

R&D 

10 

2 

11 

9 

8 

0 

2.77 

0.92 

0.85 

3 

(0.24) 

Service 

11 

1 

4 

14 

10 

0 

3.14 

0.78 

0.60 

3 

(0-52) 

Hardware 

9 

1 

10 

16 

4 

0 

2.74 

0.72 

0.51 

3 

(0.59) 

Construction 

8 

4 

10 

14 

4 

0 

2.56 

0.86 

0.75 

3 

(0.39) 

Commodity 

17 

1 

2 

13 

7 

0 

3.13 

0.74 

0.55 

3 

(0.02) 

Question  2 

CE 

3 

10 

12 

6 

9 

0 

2.38 

1.11 

1.22 

2 

0.07 

PD&RR 

3 

7 

14 

9 

7 

0 

2.43 

1.00 

1.00 

2 

(0.11) 

E&MD 

5 

4 

12 

16 

3 

1 

2.58 

0.78 

0.61 

3 

(0.70) 

PFD&OS 

6 

2 

10 

20 

2 

0 

2.65 

0.68 

0.46 

3 

(0.93) 

Question  3 

FFP 

14 

5 

9 

14 

3 

0 

2.48 

0.78 

0.61 

3 

(0.09) 

FPE 

19 

4 

7 

11 

4 

0 

2.58 

0.79 

0.63 

3 

0.30 

FPIS 

19 

4 

9 

11 

2 

0 

2.42 

0.72 

0.52 

3 

0.29 

FPR(P) 

20 

4 

9 

10 

2 

0 

2.40 

0.73 

0.54 

3 

0.40 

FPR(R) 

19 

4 

8 

11 

3 

0 

2.50 

0.76 

0.59 

3 

0.30 

FPIF 

17 

4 

8 

12 

4 

0 

2.57 

0.78 

0.60 

3 

0.12 

CPIF 

17 

5 

8 

10 

5 

0 

2.54 

0.90 

0.80 

3 

0.23 

CPAF 

17 

5 

8 

11 

4 

0 

2.50 

0.86 

0.73 

3 

0.21 

CPFF 

17 

5 

8 

11 

4 

0 

2.50 

0.86 

0.73 

3 

0.21 

T&M 

16 

5 

6 

11 

7 

0 

2.69 

0.91 

0.83 

3 

0.08 

Labor  Hour 

17 

3 

8 

11 

6 

0 

2.71 

0.75 

0.56 

3 

0.08 

Question  4 

Pre- Award 

5 

9 

13 

15 

3 

1 

2.37 

0.87 

0.75 

2 

(0.42) 

Post-Award 

5 

6 

7 

14 

12 

2 

2.93 

1.00 

1.00 

3 

(0.66) 
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Question  5 

No  Response 

#1 

#2 

#3 

#4 

#5 

Mean 

Std  Dev 

Variation 

Median 

Skew 

5a 

5 

10 

8 

18 

4 

1 

2.46 

0.94 

0.88 

3 

(0.44) 

5b 

3 

8 

17 

14 

4 

0 

2.33 

0.85 

0.73 

2 

(0.34) 

5c 

5 

13 

17 

10 

0 

1 

2.00 

0.76 

0.57 

2 

(0.32) 

5d 

3 

13 

20 

8 

2 

0 

1.98 

0.82 

0.67 

2 

0.08 

5e 

8 

6 

11 

11 

9 

0 

2.62 

0.97 

0.94 

3 

(0.32) 

5f 

3 

9 

21 

8 

5 

0 

2.21 

0.89 

0.79 

2 

0.07 

Question  6 

6a 

9 

9 

20 

5 

2 

0 

2.00 

0.78 

0.60 

2 

0.01 

6b 

5 

8 

10 

17 

5 

1 

2.54 

0.90 

0.82 

3 

(0.51) 

6c 

6 

2 

19 

11 

2 

0 

2.38 

0.69 

0.47 

2 

(0.57) 

6d 

4 

7 

16 

10 

4 

0 

2.30 

0.86 

0.74 

2 

(0.18) 

Be 

4 

6 

19 

9 

3 

0 

2.24 

0.78 

0.60 

2 

(0.27) 

6f 

9 

5 

22 

4 

0 

0 

1.97 

0.54 

0.29 

2 

(0.53) 

eg 

9 

1 

13 

13 

4 

0 

2.65 

0.74 

0.55 

3 

(0.47) 

Question  7 

7a 

4 

14 

15 

6 

2 

0 

1.89 

0.80 

0.64 

2 

0.21 

7b 

4 

13 

20 

2 

2 

0 

1.81 

0.68 

0.47 

2 

0.16 

7c 

5 

16 

17 

1 

1 

1 

1.72 

0.68 

0.46 

2 

0.38 

Question  8 

8a 

9 

5 

15 

7 

4 

0 

2.32 

0.89 

0.80 

2 

(0.00) 

8b 

9 

6 

16 

4 

5 

0 

2.26 

0.95 

0.90 

2 

0.19 

8c 

9 

11 

15 

2 

3 

0 

1.90 

0.89 

0.80 

2 

0.51 

8d 

11 

6 

16 

6 

1 

0 

2.07 

0.74 

0.55 

2 

0.00 

Question  9 

9 

14 

2 

12 

9 

3 

0 

2.50 

0.80 

0.63 

2 

0.04 

Question  1 0 

10a 

3 

2 

13 

16 

7 

0 

2.74 

0.83 

0.68 

3 

(0.70) 

10b 

9 

1 

12 

15 

3 

0 

2.65 

0.70 

0.49 

3 

(0.56) 

10c 

8 

1 

12 

17 

2 

0 

2.63 

0.65 

0.42 

3 

(0.74) 

10d 

11 

3 

11 

12 

3 

0 

2.52 

0.81 

0.66 

3 

(0.20) 
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Acquisition  Community  Group  Data 


No  Response 

#1 

#2 

#3 

#4 

#5 

Mean 

Std  Dev 

Variation 

Median 

Skew 

Question  1 

Benchmarking 

35 

3 

21 

33 

7 

2.69 

0.74 

0.55 

3 

(0.10) 

Validating 

22 

8 

26 

32 

10 

2.58 

0.88 

0.78 

3 

(0.24) 

Program  Est 

19 

12 

37 

29 

2 

2.26 

0.74 

0.54 

2 

(0.39) 

Budgeting 

19 

14 

35 

27 

5 

2.28 

0.82 

0.67 

2 

(0.22) 

ICE 

49 

9 

20 

17 

6 

2.38 

0.91 

0.82 

2 

0.63 

Software 

37 

8 

22 

19 

12 

2.57 

0.94 

0.88 

3 

0.21 

Repair  Parts 

29 

3 

16 

38 

14 

2.89 

0.76 

0.58 

3 

(0.34) 

R&D 

22 

7 

19 

23 

27 

2.92 

1.02 

1.05 

3 

(0.29) 

Service 

24 

4 

17 

27 

27 

3.03 

0.92 

0.84 

3 

(0.39) 

Hardware 

16 

7 

34 

32 

11 

2.56 

0.82 

0.68 

3 

(0.43) 

Construction 

43 

2 

21 

27 

7 

2.68 

0.73 

0.53 

3 

0.17 

Commodity 

51 

1 

7 

23 

16 

3.15 

0.77 

0.60 

3 

0.36 

Question  2 

CE 

•18 

13 

24 

28 

16 

2.58 

1.01 

1.02 

3 

(0.18) 

PD&RR 

17 

8 

33 

33 

10 

2.54 

0.83 

0.68 

3 

(0.41) 

E&MD 

15 

10 

36 

32 

7 

2.42 

0.80 

0.64 

2 

(0.42) 

PFD&OS 

14 

10 

32 

32 

11 

2.52 

0.86 

0.74 

3 

(0.42) 

Question  3 

FFP 

17 

9 

32 

32 

11 

2.54 

0.85 

0.73 

3 

(0.37) 

FPE 

33 

4 

24 

26 

13 

2.72 

0.84 

0.71 

3 

(0.04) 

FPIS 

32 

4 

22 

34 

8 

2.68 

0.76 

0.57 

3 

(0.17) 

FPR(P) 

28 

5 

25 

35 

7 

2.61 

0.76 

0.57 

3 

(0.27) 

FPR(R) 

30 

6 

23 

32 

9 

2.63 

0.81 

0.66 

3 

(0.15) 

FPIF 

29 

4 

30 

28 

7 

2.55 

0.75 

0.56 

3 

(0.19) 

CPIF 

25 

8 

22 

24 

11 

2.58 

0.86 

0.74 

3 

(0.26) 

CPAF 

24 

10 

24 

28 

13 

2.59 

0.92 

0.85 

3 

(0.17) 

CPFF 

21 

11 

34 

33 

12 

2.51 

0.91 

0.83 

3 

(0.27) 

T&M 

26 

3 

18 

27 

26 

3.03 

0.91 

0.83 

3 

(0.32) 

Labor  Hour 

26 

3 

29 

26 

24 

2.87 

0.92 

0.85 

3 

(0.29) 

Question  4 

Pre-Award 

7 

14 

30 

34 

5 

2.36 

0.79 

0.63 

2 

(0.48) 

Post-Award 

10 

5 

17 

27 

27 

3.00 

0.94 

0.88 

3 

(0.73) 
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Question  5 

No  Response 

#1 

#2 

#3 

#4 

#5 

Mean 

StdDev 

Variation 

Median 

Skew 

5a 

9 

11 

33 

34 

13 

12 

2.83 

0.88 

0.78 

3 

(0.46) 

5b 

10 

9 

44 

32 

5 

12 

2.68 

0.74 

0.54 

2 

(0.56) 

5c 

5 

6 

56 

16 

6 

8 

2.50 

0.77 

0.59 

2 

(0.02) 

5d 

8 

8 

65 

15 

4 

11 

2.47 

0.63 

0.40 

2 

(0.43) 

5e 

12 

2 

26 

38 

21 

15 

3.21 

0.78 

0.61 

3 

(0.81) 

5f 

11 

7 

52 

24 

6 

14 

2.69 

0.72 

0.51 

2 

(0.45) 

Question  6 

6a 

9 

11 

55 

20 

5 

12 

2.53 

0.72 

0.52 

2 

(0.34) 

6b 

13 

5 

27 

43 

12 

16 

3.07 

0.77 

0.60 

3 

(0.74) 

6c 

8 

10 

47 

23 

11 

11 

2.67 

0.83 

0.69 

2 

(0.25) 

6d 

12 

10 

52 

26 

4 

15 

2.64 

0.72 

0.52 

2 

(0.47) 

6e 

13 

11 

42 

27 

6 

16 

2.75 

0.78 

0.61 

2 

(0.37) 

6f 

11 

17 

55 

12 

5 

12 

2.41 

0.74 

0.55 

2 

(0.09) 

6g 

14 

5 

39 

32 

10 

17 

2.95 

0.77 

0.60 

2 

(0.52) 

Question  7 

7a 

7 

28 

54 

9 

2 

10 

2.15 

0.68 

0.46 

2 

(0.06) 

7b 

6 

27 

57 

8 

1 

9 

2.10 

0.62 

0.38 

2 

(0.28) 

7c 

7 

30 

59 

3 

1 

10 

2.05 

0.57 

0.33 

2 

(0.35) 

Question  8 

8a 

10 

16 

52 

16 

5 

13 

2.48 

0.76 

0.58 

2 

(0.19) 

8b 

10 

15 

55 

15 

5 

13 

2.48 

0.74 

0.54 

2 

(0.18) 

8c 

8 

23 

51 

5 

3 

11 

2.23 

0.65 

0.42 

2 

(0.09) 

8d 

13 

15 

60 

9 

2 

16 

2.45 

0.61 

0.38 

2 

(0.38) 

Question  9 

9 

28 

8 

43 

15 

6 

31 

3.09 

0.76 

0.58 

2 

0.02 

Question  10 

10a 

10 

6 

43 

33 

7 

14 

2.81 

0.78 

0.61 

2 

(0.45) 

10b 

16 

0 

19 

55 

8 

19 

3.27 

0.61 

0.37 

3 

(0.94) 

10c 

10 

3 

31 

45 

11 

13 

3.00 

0.72 

0.52 

3 

(0.88) 

10d 

12 

10 

36 

35 

7 

15 

2.82 

0.80 

0.63 

2 

(0.51) 
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Federal  Government  Group  Data 


No  Response 

#1 

#2 

#3 

#4 

#5 

Mean 

Std  Dev 

Variation 

Median 

Skew 

Question  1 

Benchmarking 

66 

7 

39 

55 

16 

1 

2.70 

0.81 

0.65 

3 

(0.02) 

Validating 

43 

9 

47 

67 

16 

2 

2.68 

0.81 

0.66 

3 

(0.36) 

Program  Est 

45 

21 

58 

54 

6 

0 

2.32 

0.78 

0.61 

2 

(0.25) 

Budgeting 

46 

22 

61 

47 

8 

0 

2.30 

0.80 

0.64 

2 

(0.15) 

ICE 

86 

17 

36 

33 

12 

0 

2.41 

0.91 

0.83 

2 

0.51 

Software 

68 

14 

38 

40 

23 

1 

2.65 

0.96 

0.92 

3 

0.17 

Repair  Parts 

59 

5 

28 

66 

26 

0 

2.90 

0.76 

0.58 

3 

(0.28) 

R&D 

50 

12 

35 

45 

37 

5 

2.91 

1.02 

1.04 

3 

(0.14) 

Service 

50 

7 

27 

51 

48 

1 

3.07 

0.89 

0.79 

3 

(0.38) 

Hardware 

41 

11 

53 

64 

15 

0 

2.58 

0.78 

0.61 

3 

(0.42) 

Construction 

75 

5 

37 

46 

21 

0 

2.76 

0.81 

0.66 

3 

0.11 

Commodity 

91 

2 

12 

45 

33 

1 

3.20 

0.76 

0.57 

3 

0.25 

Question  2 

CE 

43 

28 

39 

39 

33 

2 

2.59 

1.09 

1.18 

3 

0.00 

PD&RR 

39 

20 

51 

55 

18 

1 

2.51 

0.90 

0.82 

3 

(0.21) 

E&MD 

38 

14 

56 

64 

12 

0 

2.51 

0.77 

0.60 

3 

(0.45) 

PFD&OS 

39 

13 

47 

61 

24 

0 

2.66 

0.85 

0.73 

3 

(0.40) 

Question  3 

FFP 

47 

12 

49 

54 

21 

1 

2.64 

0.87 

0.76 

3 

(0.21) 

FPE 

71 

6 

36 

48 

22 

1 

2.79 

0.85 

0.72 

3 

0.07 

FPIS 

73 

5 

38 

54 

14 

0 

2.69 

0.74 

0.55 

3 

0.03 

FPR(P) 

68 

6 

42 

55 

13 

0 

2.65 

0.75 

0.56 

3 

(0.04) 

FPR(R) 

69 

6 

39 

53 

17 

0 

2.70 

0.78 

0.61 

3 

(0.01) 

FPIF 

62 

5 

49 

52 

16 

0 

2.65 

0.76 

0.57 

3 

(0.11) 

CPIF 

58 

11 

38 

55 

22 

0 

2.70 

0.86 

0.73 

3 

(0.13) 

CPAF 

60 

13 

38 

51 

21 

1 

2.67 

0.90 

0.82 

3 

(0.02) 

CPFF 

57 

15 

38 

53 

20 

1 

2.64 

0.91 

0.83 

3 

(0.05) 

T&M 

62 

4 

22 

48 

43 

5 

3.19 

0.89 

0.79 

3 

(0.18) 

Labor  Hour 

63 

4 

26 

42 

44 

5 

3.17 

0.92 

0.85 

3 

(0.12) 

Question  4 

Pre-Award 

30 

21 

56 

65 

12 

0 

2.44 

0.82 

0.68 

3 

(0.45) 

Post-Award 

37 

12 

24 

53 

51 

7 

3.12 

1.00 

1.01 

3 

(0.48) 
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Question  5 

No  Response 

#1 

#2 

#3 

#4 

#5 

Mean 

StdDev 

Variation 

Median 

Skew 

5a 

27 

19 

52 

60 

26 

0 

2.59 

0.90 

0.81 

3 

(0.43) 

5b 

29 

14 

74 

52 

15 

0 

2.44 

0.79 

0.62 

2 

(0.41) 

5c 

23 

31 

89 

33 

8 

0 

2.11 

0.76 

0.58 

2 

(0.18) 

5d 

25 

23 

91 

34 

11 

0 

2.21 

0.77 

0.59 

2 

(0.22) 

5e 

43 

5 

34 

66 

36 

0 

2.94 

0.80 

0.64 

3 

(0.54) 

5f 

33 

13 

89 

37 

12 

0 

2.32 

0.74 

0.55 

2 

(0.30) 

Question  6 

6a 

33 

22 

88 

31 

10 

0 

2.19 

0.76 

0.58 

2 

(0.18) 

6b 

33 

10 

47 

71 

23 

0 

2.71 

0.80 

0.64 

3 

(0.59) 

6c 

33 

16 

81 

37 

17 

0 

2.36 

0.82 

0.67 

2 

(0.21) 

6d 

32 

20 

77 

44 

11 

0 

2.30 

0.79 

0.62 

2 

(0.29) 

6e 

33 

22 

68 

47 

14 

0 

2.35 

0.84 

0.70 

2 

(0.24) 

6f 

34 

32 

87 

23 

8 

0 

2.05 

0.76 

0.58 

2 

(0.04) 

6g 

43 

11 

58 

54 

18 

0 

2.56 

0.81 

0.66 

3 

(0.31) 

Question  7 

7a 

22 

60 

86 

12 

4 

0 

1.75 

0.69 

0.48 

2 

0.12 

7b 

21 

54 

97 

10 

2 

0 

1.75 

0.62 

0.38 

2 

(0.21) 

7c 

22 

62 

94 

4 

2 

0 

1.67 

0.59 

0.35 

2 

(0.18) 

Question  8 

8a 

33 

27 

80 

30 

14 

0 

2.21 

0.84 

0.71 

2 

(0.04) 

8b 

33 

29 

80 

29 

13 

0 

2.17 

0.84 

0.70 

2 

(0.02) 

8c 

29 

45 

89 

14 

7 

0 

1.89 

0.74 

0.55 

2 

0.10 

8d 

37 

29 

89 

21 

8 

0 

2.05 

0.74 

0.55 

2 

(0.04) 

Question  9 

9 

56 

13 

69 

34 

11 

1 

2.36 

0.81 

0.65 

2 

0.05 

Question  10 

10a 

29 

9 

65 

61 

20 

0 

2.59 

0.78 

0.62 

3 

(0.54) 

10b 

46 

1 

41 

75 

19 

2 

2.86 

0.71 

0.50 

3 

(0.49) 

10c 

33 

6 

57 

67 

20 

1 

2.69 

0.77 

0.60 

3 

(0.55) 

10d 

36 

19 

54 

61 

13 

1 

2.48 

0.85 

0.72 

3 

(0-31) 
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Prepare  Estimates  Group  Data 


No  Response 

#1 

#2 

#3 

#4 

#5 

Mean 

StdDev 

Variation 

Median 

Skew 

Question  1 

Benchmarking 

10 

8 

7 

7 

5 

0 

2.33 

1.09 

1.19 

2 

0.25 

Validating 

1 

10 

12 

12 

2 

0 

2.17 

0.90 

0.81 

2 

(0.03) 

Program  Est 

2 

13 

11 

10 

1 

0 

1.97 

0.88 

0.77 

2 

0.09 

Budgeting 

2 

12 

14 

8 

1 

0 

1.94 

0.83 

0.68 

2 

0.12 

ICE 

9 

9 

6 

11 

2 

0 

2.21 

0.98 

0.95 

2 

0.08 

Software 

9 

10 

4 

11 

3 

0 

2.25 

1.06 

1.12 

3 

0.15 

Repair  Parts 

11 

3 

6 

10 

7 

0 

2.81 

0.96 

0.92 

3 

(0.15) 

R&D 

10 

8 

6 

9 

4 

0 

2.33 

1.05 

1.11 

2 

0.18 

Service 

8 

2 

4 

14 

9 

0 

3.03 

0.85 

0.72 

3 

(0.65) 

Hardware 

7 

6 

5 

15 

4 

0 

2.57 

0.96 

0.91 

3 

(0.37) 

Construction 

11 

6 

4 

12 

4 

0 

2.54 

1.01 

1.02 

3 

(0.00) 

Commodity 

15 

2 

2 

11 

7 

0 

3.05 

0.88 

0.77 

3 

0.01 

Question  2 

CE 

2 

17 

10 

5 

3 

0 

1.83 

0.97 

0.94 

2 

0.74 

PD&RR 

3 

14 

12 

5 

3 

0 

1.91 

0.95 

0.90 

2 

0.52 

E&MD 

3 

7 

9 

17 

1 

0 

2.35 

0.84 

0.70 

3 

(0.65) 

PFD&OS 

5 

3 

7 

16 

6 

0 

2.78 

0.84 

0.70 

3 

(0.76) 

Question  3 

FFP 

10 

5 

9 

9 

4 

0 

2.44 

0.96 

0.91 

2 

0.01 

FPE 

17 

5 

6 

4 

5 

0 

2.45 

1.12 

1.25 

2 

0.68 

FPIS 

17 

5 

7 

6 

2 

0 

2.25 

0.94 

0.89 

2 

0.61 

FPR(P) 

16 

5 

7 

7 

2 

0 

2.29 

0.93 

0.87 

2 

0.49 

FPR(R) 

16 

5 

7 

6 

3 

0 

2.33 

0.99 

0.98 

2 

0.55 

FPIF 

13 

7 

7 

6 

4 

0 

2.29 

1.06 

1.12 

2 

0.44 

CPIF 

13 

5 

10 

5 

4 

0 

2.33 

0.99 

0.97 

2 

0.37 

CPAF 

14 

5 

10 

5 

3 

0 

2.26 

0.94 

0.89 

2 

0.43 

CPFF 

14 

5 

8 

6 

4 

0 

2.39 

1.01 

1.02 

2 

0.39 

T&M 

14 

4 

5 

6 

8 

0 

2.78 

1.10 

1.21 

3 

0.22 

Labor  Hour 

14 

2 

6 

8 

7 

0 

2.87 

0.95 

0.90 

3 

0.07 

Question  4 

Pre-Award 

3 

9 

12 

12 

1 

0 

2.15 

0.84 

0.71 

2 

(0.28) 

Post-Award 

4 

5 

10 

12 

6 

0 

2.58 

0.94 

0.88 

3 

(0.45) 
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Question  5 

No  Response 

#1 

#2 

#3 

#4 

#5 

Mean 

Std  Dev 

Variation 

Median 

Skew 

5a 

4 

6 

12 

12 

3 

0 

2.36 

0.88 

0.78 

2 

(0.35) 

5b 

2 

6 

15 

13 

1 

0 

2.26 

0.77 

0.59 

2 

(0.51) 

5c 

2 

13 

12 

10 

0 

0 

1.91 

0.81 

0.65 

2 

(0.07) 

5d 

1 

16 

15 

4 

1 

0 

1.72 

0.77 

0.59 

2 

0.68 

5e 

7 

4 

12 

10 

4 

0 

2.47 

0.88 

0.78 

2 

(0.26) 

5f 

4 

6 

22 

5 

0 

0 

1.97 

0.58 

0.33 

2 

(0.74) 

Question  6 

6a 

6 

9 

16 

5 

1 

0 

1.94 

0.76 

0.58 

2 

(0.01) 

6b 

4 

6 

8 

13 

6 

0 

2.58 

0.99 

0.97 

3 

(0.42) 

6c 

8 

6 

15 

7 

1 

0 

2.10 

0.76 

0.58 

2 

(0.14) 

6d 

6 

8 

14 

8 

1 

0 

2.06 

0.80 

0.64 

2 

(0.14) 

6e 

8 

7 

13 

7 

2 

0 

2.14 

0.86 

0.74 

2 

0.04 

6f 

5 

6 

19 

7 

0 

0 

2.03 

0.64 

0.41 

2 

(0.59) 

6g 

6 

3 

11 

13 

4 

0 

2.58 

0.83 

0.70 

3 

(0.51) 

Question  7 

7a 

3 

11 

15 

6 

2 

0 

1.97 

0.86 

0.73 

2 

0.22 

7b 

3 

11 

20 

2 

1 

0 

1.79 

0.68 

0.46 

2 

0.12 

7c 

4 

12 

19 

1 

1 

0 

1.73 

0.66 

0.44 

2 

0.17 

Question  8 

8a 

8 

5 

12 

10 

2 

0 

2.31 

0.83 

0.70 

2 

(0.20) 

8b 

9 

4 

14 

7 

3 

0 

2.32 

0.85 

0.72 

2 

(0.04) 

8c 

9 

6 

17 

4 

1 

0 

2.00 

0.71 

0.50 

2 

(0.04) 

8d 

10 

3 

19 

4 

1 

0 

2.11 

0.63 

0.40 

2 

(0.18) 

Question  9 

9 

10 

3 

14 

9 

1 

0 

2.30 

0.70 

0.49 

2 

(0.24) 

Question  1 0 

10a 

2 

1 

13 

13 

8 

0 

2.80 

0.82 

0.67 

3 

(0.66) 

10b 

5 

1 

12 

13 

5 

1 

2.78 

0.86 

0.73 

3 

(0.48) 

10c 

3 

1 

10 

19 

4 

0 

2.76 

0.69 

0.47 

3 

(1.12) 

10d 

4 

3 

17 

12 

1 

0 

2.33 

0.68 

0.46 

2 

(0.73) 
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Negotiation  Preparation  Group  Data 


No  Response 

#1 

#2 

#3 

#4 

#5 

Mean 

Std  Dev 

Variation 

Median 

Skew 

Question  1 

Benchmarking 

33 

4 

23 

35 

5 

1 

2.65 

0.76 

0.58 

3 

(0.14) 

Validating 

23 

3 

29 

41 

5 

0 

2.62 

0.66 

0.44 

3 

(0.60) 

Program  Est 

24 

13 

37 

25 

2 

0 

2.21 

0.74 

0.55 

2 

(0.24) 

Budgeting 

27 

14 

37 

20 

3 

0 

2.16 

0.77 

0.60 

2 

(0.05) 

ICE 

44 

6 

21 

24 

6 

0 

2.53 

0.82 

0.67 

3 

0.27 

Software 

38 

10 

22 

20 

11 

0 

2.51 

0.96 

0.92 

2 

0.25 

Repair  Parts 

34 

3 

17 

37 

10 

0 

2.81 

0.74 

0.54 

3 

(0.22) 

R&D 

26 

9 

22 

24 

18 

2 

2.76 

1.03 

1.06 

3 

(0.09) 

Service 

28 

4 

13 

32 

23 

1 

3.05 

0.87 

0.76 

3 

(0.37) 

Hardware 

20 

6 

27 

39 

9 

0 

2.63 

0.78 

0.60 

3 

(0.54) 

Construction 

45 

3 

16 

24 

13 

0 

2.84 

0.84 

0.71 

3 

0.22 

Commodity 

50 

1 

5 

23 

21 

1 

3.31 

0.75 

0.57 

3 

0.23 

Question  2 

CE 

23 

15 

25 

22 

14 

2 

2.53 

1.07 

1.15 

2 

0.07 

PD&RR 

21 

13 

24 

34 

8 

1 

2.50 

0.92 

0.85 

3 

(0.21) 

E&MD 

18 

9 

34 

35 

5 

0 

2.43 

0.76 

0.58 

2 

(0.50) 

PFD&OS 

20 

9 

26 

35 

11 

0 

2.59 

0.85 

0.72 

3 

(0.41) 

Question  3 

FFP 

18 

7 

27 

37 

12 

0 

2.65 

0.83 

0.69 

3 

(0.52) 

FPE 

35 

3 

17 

33 

13 

0 

2.85 

0.78 

0.61 

3 

(0.16) 

FPIS 

37 

2 

20 

36 

6 

0 

2.72 

0.67 

0.45 

3 

(0.15) 

FPR(P) 

32 

3 

23 

36 

7 

0 

2.68 

0.71 

0.51 

3 

(0.27) 

FPR(R) 

34 

3 

21 

33 

10 

0 

2.75 

0.76 

0.58 

3 

(0.17) 

FPIF 

31 

2 

25 

34 

9 

0 

2.71 

0.72 

0.52 

3 

(0.28) 

CPIF 

28 

6 

22 

33 

12 

0 

2.70 

0.84 

0.70 

3 

(0.26) 

CPAF 

28 

7 

22 

33 

11 

0 

2.66 

0.85 

0.72 

3 

(0.24) 

CPFF 

25 

8 

22 

34 

12 

0 

2.66 

0.87 

0.75 

3 

(0.30) 

T&M 

29 

4 

14 

30 

23 

1 

3.04 

0.89 

0.79 

3 

(0.32) 

Labor  Hour 

32 

3 

17 

24 

24 

1 

3.04 

0.91 

0.82 

3 

(0.19) 

Question  4 

Pre-Award 

13 

9 

32 

38 

9 

0 

2.53 

0.81 

0.66 

3 

(0.58) 

Post-Award 

19 

4 

20 

27 

28 

3 

3.07 

0.96 

0.91 

3 

(0.51) 
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Question  5 

No  Response 

#1 

#2 

#3 

#4 

#5 

Mean 

Std  Dev 

Variation 

Median 

Skew 

5a 

16 

8 

36 

30 

11 

0 

2.52 

0.83 

0.70 

2 

(0.41) 

5b 

14 

9 

41 

31 

6 

0 

2.39 

0.76 

0.58 

2 

(0.49) 

5c 

12 

20 

45 

19 

5 

0 

2.10 

0.81 

0.65 

2 

(0.10) 

5d 

15 

12 

56 

14 

4 

0 

2.12 

0.69 

0.47 

2 

(0.28) 

5e 

22 

2 

15 

41 

21 

0 

3.03 

0.75 

0.56 

3 

(0.70) 

5f 

15 

7 

50 

23 

6 

0 

2.33 

0.72 

0.52 

2 

(0.41) 

Question  6 

6a 

22 

13 

46 

16 

4 

0 

2.14 

0.74 

0.55 

2 

(0.13) 

6b 

18 

7 

25 

36 

15 

0 

2.71 

0.86 

0.74 

3 

(0.52) 

6c 

14 

12 

47 

20 

8 

0 

2.28 

0.81 

0.66 

2 

(0.20) 

6d 

18 

8 

41 

26 

8 

0 

2.41 

0.79 

0.63 

2 

(0.34) 

6e 

19 

11 

35 

29 

7 

0 

2.39 

0.82 

0.68 

2 

(0.31) 

6f 

18 

17 

49 

12 

5 

0 

2.06 

0.77 

0.59 

2 

(0.03) 

6g 

19 

8 

36 

29 

9 

0 

2.48 

0.81 

0.66 

2 

(0.36) 

Question  7 

7a 

12 

30 

50 

7 

2 

0 

1.79 

0.68 

0.46 

2 

0.00 

7b 

12 

30 

53 

5 

1 

0 

1.74 

0.61 

0.37 

2 

(0.23) 

7c 

14 

31 

52 

3 

1 

0 

1.70 

0.59 

0.35 

2 

(0.23) 

Question  8 

8a 

15 

15 

49 

14 

8 

0 

2.17 

0.82 

0.68 

2 

(0.03) 

8b 

16 

13 

52 

15 

5 

0 

2.14 

0.74 

0.54 

2 

(0.19) 

8c 

15 

25 

49 

7 

5 

0 

1.91 

0.77 

0.60 

2 

0.19 

8d 

17 

15 

52 

11 

6 

0 

2.10 

0.77 

0.59 

2 

(0.04) 

Question  9 

9 

30 

6 

37 

20 

8 

0 

2.42 

0.80 

0.63 

2 

(0.04) 

Question  1 0 

10a 

15 

3 

38 

38 

7 

0 

2.57 

0.69 

0.48 

3 

(0.73) 

10b 

21 

0 

24 

43 

13 

0 

2.86 

0.67 

0.44 

3 

(0.73) 

10c 

18 

4 

26 

42 

11 

0 

2.72 

0.75 

0.56 

3 

(0.68) 

10d 

19 

13 

35 

28 

6 

0 

2.33 

0.83 

0.68 

2 

(0.27) 
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Review  or  Audit  Estimates  Group  Data 


No  Response 

#1 

#2 

#3 

#4 

#5 

Mean 

Std  Dev 

Variation 

Median 

Skew 

Question  1 

Benchmarking 

33 

6 

19 

27 

15 

1 

2.79 

0.93 

0.87 

3 

(0.04) 

Validating 

18 

9 

25 

36 

10 

3 

2.67 

0.95 

0.89 

3 

(0.27) 

Program  Est 

28 

10 

28 

31 

4 

0 

2.40 

0.79 

0.62 

2 

(0.19) 

Budgeting 

29 

9 

33 

26 

4 

0 

2.35 

0.77 

0.59 

2 

(0.14) 

ICE 

52 

8 

15 

18 

8 

0 

2.53 

0.95 

0.90 

3 

0.63 

Software 

41 

11 

14 

23 

10 

2 

2.63 

1.06 

1.13 

3 

0.38 

Repair  Parts 

34 

2 

16 

32 

17 

0 

2.96 

0.78 

0.61 

3 

(0.22) 

R&D 

28 

4 

16 

25 

25 

3 

3.10 

0.97 

0.94 

3 

(0.26) 

Service 

31 

2 

12 

26 

30 

0 

3.20 

0.82 

0.67 

3 

(0.37) 

Hardware 

25 

7 

29 

31 

9 

0 

2.55 

0.82 

0.67 

3 

(0.28) 

Construction 

39 

2 

17 

30 

13 

0 

2.87 

0.77 

0.60 

3 

(0.03) 

Commodity 

52 

0 

5 

24 

19 

1 

3.33 

0.68 

0.46 

3 

0.28 

Question  2 

CE 

27 

17 

21 

19 

17 

0 

2.49 

1.07 

1.15 

2 

0.11 

PD&RR 

22 

13 

25 

32 

8 

1 

2.48 

0.93 

0.86 

3 

(0.16) 

E&MD 

0 

6 

27 

38 

5 

0 

2.55 

0.72 

0.52 

3 

(0.46) 

PFD&OS 

28 

5 

25 

33 

10 

0 

2.66 

0.80 

0.63 

3 

(0.27) 

Question  3 

FFP 

30 

6 

30 

24 

11 

0 

2.56 

0.85 

0.72 

2 

(0.08) 

FPE 

42 

3 

24 

21 

11 

0 

2.68 

0.83 

0.69 

3 

0.20 

FPIS 

43 

3 

22 

25 

8 

0 

2.66 

0.78 

0.61 

3 

0.17 

FPR(P) 

42 

3 

26 

23 

7 

0 

2.58 

0.76 

0.58 

3 

0.17 

FPR(R) 

43 

3 

23 

23 

9 

0 

2.66 

0.80 

0.64 

3 

0.20 

FPIF 

35 

5 

27 

25 

9 

0 

2.58 

0.82 

0.67 

3 

0.01 

CPIF 

33 

4 

20 

32 

12 

0 

2.76 

0.81 

0.65 

3 

(0.17) 

CPAF 

36 

5 

17 

32 

11 

0 

2.75 

0.82 

0.68 

3 

(0.07) 

CPFF 

35 

5 

15 

35 

11 

0 

2.79 

0.81 

0.65 

3 

(0.13) 

T&M 

36 

3 

9 

26 

25 

2 

3.22 

0.89 

0.78 

3 

(0.15) 

Labor  Hour 

38 

2 

10 

24 

25 

2 

3.24 

0.87 

0.75 

3 

(0.09) 

Question  4 

Pre-Award 

19 

12 

35 

31 

4 

0 

2.33 

0.78 

0.61 

2 

(0.38) 

Post-Award 

23 

3 

15 

25 

28 

7 

3.27 

1.00 

0.99 

3 

(0.43) 
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Question  5 

No  Response 

#1 

#2 

#3 

#4 

#5 

Mean 

Std  Dev 

Variation 

Median 

Skew 

5a 

14 

11 

32 

30 

14 

0 

2.54 

0.91 

0.82 

3 

(0.37) 

5b 

12 

9 

51 

24 

5 

0 

2.28 

0.72 

0.52 

2 

(0.44) 

5c 

14 

18 

47 

19 

3 

0 

2.08 

0.75 

0.56 

2 

(0.24) 

5d 

14 

17 

39 

26 

5 

0 

2.22 

0.82 

0.68 

2 

(0.25) 

5e 

24 

1 

17 

38 

21 

0 

3.03 

0.74 

0.54 

3 

(0.62) 

5f 

15 

10 

48 

21 

7 

0 

2.29 

0.78 

0.60 

2 

(0.28) 

Question  6 

6a 

21 

12 

39 

24 

5 

0 

2.28 

0.79 

0.62 

2 

(0.22) 

6b 

16 

4 

28 

37 

16 

0 

2.76 

0.81 

0.65 

3 

(0.64) 

6c 

19 

12 

40 

19 

11 

0 

2.35 

0.89 

0.79 

2 

(0.09) 

6d 

19 

14 

41 

23 

4 

0 

2.21 

0.78 

0.60 

2 

(0.24) 

6e 

20 

16 

31 

29 

5 

0 

2.28 

0.85 

0.72 

2 

(0.21) 

6f 

22 

19 

41 

16 

3 

0 

2.04 

0.77 

0.59 

2 

(0.03) 

6g 

23 

7 

29 

33 

9 

0 

2.56 

0.81 

0.66 

3 

(0.36) 

Question  7 

7a 

14 

36 

42 

6 

3 

0 

1.72 

0.74 

0.54 

2 

0.33 

7b 

14 

32 

50 

4 

1 

0 

1.70 

0.61 

0.37 

2 

(0.16) 

7c 

16 

36 

45 

3 

1 

0 

1.64 

0.61 

0.37 

2 

(0.03) 

Question  8 

8a 

20 

17 

35 

18 

11 

0 

2.28 

0.95 

0.89 

2 

0.04 

8b 

21 

20 

31 

19 

10 

0 

2.24 

0.96 

0.93 

2 

0.10 

8c 

17 

26 

43 

10 

5 

0 

1.93 

0.81 

0.66 

2 

0.21 

8d 

18 

17 

49 

13 

4 

0 

2.05 

0.74 

0.55 

2 

(0.09) 

Question  9 

9 

33 

9 

36 

18 

4 

1 

2.29 

0.82 

0.68 

2 

0.18 

Question  10 

10a 

17 

5 

40 

30 

9 

0 

2.51 

0.76 

0.58 

2 

(0.48) 

10b 

26 

1 

18 

46 

9 

1 

2.88 

0.67 

0.45 

3 

(0.55) 

10c 

19 

4 

28 

40 

9 

1 

2.70 

0.78 

0.60 

3 

(0.54) 

10d 

20 

8 

37 

29 

6 

1 

2.44 

0.82 

0.67 

2 

(0.26) 
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Set  Corporate  Policy  Group  Data 


No  Response 

#1 

#2 

#3 

#4 

#5 

Mean 

Std  Dev 

Variance 

Median 

Skew 

Question  1 

Benchmarking 

0 

3 

2 

0 

0 

0 

1.40 

0.49 

0.24 

1 

0.61 

Validating 

0 

2 

3 

0 

0 

0 

1.60 

0.49 

0.24 

2 

(0.61) 

Program  Est 

0 

3 

1 

1 

0 

0 

1.60 

0.80 

0.64 

1 

1.26 

Budgeting 

0 

2 

3 

0 

0 

0 

1.60 

0.49 

0.24 

2 

(0.61) 

ICE 

1 

2 

0 

2 

0 

0 

2.00 

1.00 

1.00 

2 

0.17 

Software 

0 

3 

1 

1 

0 

0 

1.60 

0.80 

0.64 

1 

1.26 

Repair  Parts 

1 

2 

1 

0 

2 

0 

2.40 

1.30 

1.69 

3 

(0.05) 

R&D 

1 

1 

1 

1 

1 

0 

2.50 

1.12 

1.25 

3 

0.00 

Service 

1 

0 

0 

3 

1 

0 

3.25 

0.43 

0.19 

3 

(1.75) 

Hardware 

1 

2 

0 

1 

1 

0 

2.25 

1.30 

1.69 

2 

0.52 

Construction 

0 

1 

0 

2 

2 

0 

3.00 

1.10 

1.20 

3 

(1.36) 

Commodity 

1 

0 

0 

2 

2 

0 

3.50 

0.50 

0.25 

4 

(1.74) 

Question  2 

CE 

0 

4 

1 

0 

0 

0 

1.20 

0.40 

0.16 

1 

2.24 

PD&RR 

0 

3 

2 

0 

0 

0 

1.40 

0.49 

0.24 

1 

0.61 

E&MD 

0 

2 

2 

1 

0 

0 

1.80 

0.75 

0.56 

2 

0.51 

PFD&OS 

0 

2 

2 

1 

0 

0 

1.80 

0.80 

0.64 

2 

0.05 

Question  3 

FFP 

1 

1 

2 

0 

0 

2.25 

0.83 

0.69 

3 

(0.54) 

FPE 

2 

2 

0 

0 

2.00 

0.82 

0.67 

2 

0.54 

FPIS 

2 

1 

0 

0 

2.00 

0.82 

0.67 

2 

0.54 

FPR(P) 

2 

1 

0 

0 

2.00 

0.82 

0.67 

2 

0.54 

FPR(R) 

2 

1 

0 

0 

2.00 

0.82 

0.67 

2 

0.54 

FPIF 

2 

1 

0 

0 

2.00 

0.82 

0.67 

2 

0.54 

CPIF 

2 

2 

0 

0 

0 

1.67 

0.94 

0.89 

1 

1.36 

CPAF 

2 

2 

0 

0 

0 

1.67 

0.94 

0.89 

1 

1.36 

CPFF 

2 

2 

0 

0 

0 

1.67 

0.94 

0.89 

1 

1.36 

T&M 

2 

2 

0 

0 

0 

1.67 

0.94 

0.89 

1 

1.36 

Labor  Hour 

3 

0 

0 

2 

0 

0 

3.00 

0.00 

0.00 

3 

0.61 

Question  4 

Pre- Award 

0 

2 

3 

0 

0 

0 

1.60 

0.49 

0.24 

2 

(0.61) 

Post-Award 

0 

1 

3 

1 

0 

0 

2.00 

0.63 

0.40 

2 

0.00 
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Question  5 

No  Response 

#1 

#? 

#3 

#4 

#5 

Mean 

Std  Dev 

Variance 

Median 

Skew 

5a 

0 

3 

2 

0 

0 

0 

1.40 

0.49 

0.24 

1 

0.61 

5b 

0 

3 

2 

0 

0 

0 

1.40 

0.49 

0.24 

1 

0.61 

5c 

0 

4 

1 

0 

0 

0 

1.20 

0.40 

0.16 

1 

2.24 

5d 

0 

4 

1 

0 

0 

0 

1.20 

0.40 

0.16 

1 

2.24 

5e 

2 

0 

2 

1 

0 

0 

2.33 

0.47 

0.22 

2 

(0.17) 

5f 

1 

3 

1 

0 

0 

0 

1.25 

0.43 

0.19 

1 

0.00 

Question  6 

6a 

1 

3 

2 

0 

0 

0 

1.40 

0.43 

0.19 

1 

0.00 

5b 

0 

2 

1 

1 

0 

2.20 

1.17 

1.36 

2 

0.54 

6c 

0 

2 

2 

0 

0 

1.80 

0.75 

0.56 

2 

0.51 

6d 

0 

3 

1 

0 

0 

1.60 

0.80 

0.64 

1 

1.26 

5e 

0 

3 

1 

0 

0 

1.60 

0.80 

0.64 

1 

1.26 

6f 

0 

2 

2 

0 

0 

1.80 

0.75 

0.56 

2 

0.51 

6g 

1 

1 

2 

0 

0 

2.00 

0.83 

0.69 

3 

(0.54) 

Question  7 

7a 

0 

2 

1 

2 

0 

0 

2.00 

0.89 

0.80 

2 

0.00 

7b 

0 

3 

2 

0 

0 

0 

1.40 

0.49 

0.24 

1 

0.61 

7c 

1 

3 

1 

0 

0 

0 

1.25 

0.43 

0.19 

1 

0.00 

Question  8 

8a 

0 

1 

3 

0 

1 

0 

2.20 

0.98 

0.96 

2 

1.29 

8b 

1 

1 

2 

0 

1 

0 

2.25 

1.09 

1.19 

2 

0.55 

8c 

1 

2 

3 

0 

0 

0 

1.60 

0.43 

0.19 

2 

(1.26) 

8d 

1 

0 

3 

1 

0 

0 

2.25 

0.43 

0.19 

2 

(1.29) 

Question  9 

9 

1 

0 

3 

1 

0 

0 

2.25 

0.43 

0.19 

2 

(1.29) 

Question  10 

10a 

0 

0 

4 

1 

0 

0 

2.20 

0.40 

0.16 

2 

2.24 

10b 

0 

0 

3 

2 

0 

0 

2.40 

0.49 

0.24 

2 

0.61 

10c 

0 

1 

1 

3 

0 

0 

2.40 

0.80 

0.64 

3 

(1-26) 

10d 

1 

1 

3 

0 

0 

0 

1.75 

0.43 

0.19 

2 

(1.26) 
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Federal  Government  Policy  Group  Data 


No  Response 

#1 

#2 

#3 

#4 

#5 

Mean 

Std  Dev 

Variation 

Median 

Skew 

Question  1 

Benchmarking 

8 

2 

10 

6 

2 

0 

2.40 

0.80 

0.64 

2 

(0.08) 

Validating 

6 

3 

8 

7 

4 

0 

2.55 

0.94 

0.88 

3 

(0.19) 

Program  Est 

4 

7 

9 

7 

1 

0 

2.08 

0.86 

0.74 

2 

(0.08) 

Budgeting 

5 

5 

9 

8 

1 

0 

2.22 

0.83 

0.69 

2 

(0.24) 

ICE 

11 

6 

6 

3 

2 

0 

2.06 

1.00 

1.00 

2 

0.71 

Software 

6 

4 

6 

7 

4 

1 

2.64 

1.11 

1.23 

3 

0.01 

Repair  Parts 

7 

0 

4 

12 

5 

0 

3.05 

0.65 

0.43 

3 

(0.69) 

R&D 

7 

1 

5 

8 

5 

2 

3.10 

1.02 

1.04 

3 

(0.23) 

Service 

5 

1 

6 

9 

7 

0 

2.96 

0.86 

0.74 

3 

(0.67) 

Hardware 

3 

3 

7 

12 

3 

0 

2.60 

0.85 

0.72 

3 

(0.69) 

Construction 

5 

0 

8 

11 

4 

0 

2.83 

0.70 

0.49 

3 

(0.77) 

Commodity 

11 

0 

2 

7 

8 

0 

3.35 

0.68 

0.46 

3 

(0.19) 

Question  2 

CE 

4 

6 

6 

8 

4 

0 

2.42 

1.04 

1.08 

3 

(0.14) 

PD&RR 

4 

3 

11 

7 

3 

0 

2.42 

0.86 

0.74 

2 

(0.29) 

E&MD 

4 

2 

12 

8 

2 

0 

2.42 

0.76 

0.58 

2 

(0.49) 

PFD&OS 

3 

2 

11 

7 

5 

0 

2.60 

0.89 

0.80 

2 

(0.40) 

Question  3 

FFP 

5 

2 

9 

9 

3 

0 

2.57 

0.82 

0.68 

3 

(0.46) 

FPE 

7 

0 

7 

11 

3 

0 

2.81 

0.66 

0.44 

3 

(0.58) 

FPIS 

6 

1 

8 

10 

3 

0 

2.68 

0.76 

0.58 

3 

(0.53) 

FPR(P) 

6 

1 

7 

11 

3 

0 

2.73 

0.75 

0.56 

3 

(0.59) 

FPR(R) 

6 

1 

8 

10 

3 

0 

2.68 

0.76 

0.58 

3 

(0.53) 

FPIF 

6 

0 

10 

9 

3 

0 

2.68 

0.70 

0.49 

3 

(0.55) 

CPIF 

6 

2 

6 

10 

4 

0 

2.73 

0.86 

0.74 

3 

(0.46) 

CPAF 

7 

2 

7 

8 

3 

1 

2.71 

0.98 

0.97 

3 

(0.07) 

CPFF 

7 

2 

7 

8 

3 

1 

2.71 

0.98 

0.97 

3 

(0.07) 

T&M 

6 

0 

2 

13 

6 

1 

3.27 

0.69 

0.47 

3 

(0.82) 

Labor  Hour 

6 

0 

3 

10 

8 

1 

3.32 

0.76 

0.58 

3 

(0.76) 

Question  4 

Pre-Award 

4 

6 

9 

9 

0 

0 

2.13 

0.78 

0.61 

2 

(0.43) 

Post-Award 

5 

1 

6 

11 

5 

0 

2.87 

0.80 

0.64 

3 

(0.72) 
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Question  5 

No  Response 

#1 

#2 

#3 

#4 

#5 

Mean 

Std  Dev 

Variation 

Median 

Skew 

5a 

2 

3 

10 

9 

4 

0 

2.54 

0.89 

0.79 

3 

(0.42) 

5b 

3 

1 

13 

8 

3 

0 

2.52 

0.75 

0.57 

2 

(0.54) 

5c 

2 

7 

14 

3 

2 

0 

2.00 

0.83 

0.69 

2 

0.30 

5d 

3 

5 

13 

5 

2 

0 

2.16 

0.83 

0.69 

2 

(0.05) 

5e 

5 

1 

7 

10 

5 

0 

2.83 

0.82 

0.67 

3 

(0.64) 

5f 

5 

3 

14 

4 

2 

0 

2.22 

0.78 

0.60 

2 

(0.13) 

Question  6 

6a 

4 

4 

14 

5 

1 

0 

2.13 

0.73 

0.53 

2 

(0.29) 

6b 

5 

1 

9 

9 

4 

0 

2.70 

0.80 

0.65 

3 

(0.54) 

6c 

2 

3 

12 

9 

2 

0 

2.38 

0.79 

0.62 

2 

(0.46) 

Bd 

3 

2 

11 

8 

4 

0 

2.56 

0.85 

0.73 

2 

(0.45) 

6e 

2 

3 

11 

8 

4 

0 

2.50 

0.89 

0.79 

2 

(0.33) 

6f 

3 

7 

15 

2 

1 

0 

1.88 

0.71 

0.51 

2 

0.07 

6g 

4 

1 

11 

9 

3 

0 

2.58 

0.76 

0.58 

3 

(0.60) 

Question  7 

7a 

3 

9 

12 

3 

1 

0 

1.84 

0.78 

0.61 

2 

0.25 

7b 

2 

9 

14 

2 

1 

0 

1.81 

0.73 

0.54 

2 

0.33 

7c 

3 

10 

13 

1 

1 

0 

1.72 

0.72 

0.52 

2 

0.41 

Question  8 

8a 

3 

2 

19 

2 

2 

0 

2.16 

0.67 

0.45 

2 

(0.14) 

8b 

2 

1 

20 

2 

3 

0 

2.27 

0.71 

0.50 

2 

0.09 

8c 

2 

5 

18 

2 

1 

0 

1.96 

0.65 

0.42 

2 

(0.08) 

8d 

3 

2 

17 

5 

1 

0 

2.20 

0.63 

0.40 

2 

(0.54) 

Question  9 

9 

8 

2 

11 

4 

2 

1 

2.45 

0.97 

0.95 

2 

0.31 

Question  1 0 

10a 

4 

1 

13 

8 

2 

0 

2.46 

0.71 

0.50 

2 

(0.59) 

10b 

4 

0 

7 

11 

5 

1 

3.00 

0.82 

0.67 

3 

(0.69) 

10c 

4 

0 

9 

6 

8 

1 

3.04 

0.93 

0.87 

3 

(0.54) 

10d 

6 

4 

6 

9 

2 

1 

2.55 

1.03 

1.07 

3 

0.00 
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